Skip to content

Draft: Prototype of MetaReader caching

Tadej Novak requested to merge tadej/athena:metareader-cache into main

Overlay is very heavy on metadata peeking as it needs to load (almost) the whole metadata so it uses full mode of MetaReader (but also peeker automatically with flags autoconfiguration). For development this represented a lot of time.

This is a prototype of MetaReader caching that would only be used for development and disabled by default (enabled using the ATHENA_CACHE_METADATA environment variable). For now metadata is dumped in a pickle file per file guid.

Additionally the support for full mode in the DynamicallyLoadMetadata was also added. OverlayMetadata is also migrated to use this as separate implementation was used before.

The open questions are:

  • Should we proceed with this?
  • What format would be good for the cache? Pickle is probably the safest as it preserves types.

/cc @amete @maszyman @jchapman @fwinkl @tbold @jcatmore

Merge request reports