Explicit TQ(Sample)Folder variant support
As datasets increase also the number of systematic variations tends to increase as more detailed/higher granular descriptions are needed to provide statistical models adequate for the available statistical precision. Currently, we create an almost full copy of the SampleFolder structure for each systematic variation which contributes to the memory consumption by a significant amount and leads to issues like reported in, e.g., #316 .
Suggestion: we should add support for some sort of "variants" of folders which ideally (i.e. not sure yet how this would work technically)
- keep support for the current "path" based organization from a user perspective
- create only a lightweight layer on top of individual elements of the folder tree (e.g. the channel/systematics layer), while again sharing the lower level contents (subfolders,...)
- inherently this would require also a support for such variants that's "transient" through folder levels in order to, e.g., still tell histograms of different channels/systematic variations appart.
- this may be considered as a suggestion similar to shallow copies (I guess "variant" is just a more user-perspective driven name).
Another though that's currently on my mind in this regard is about keeping track of tags which are being read in between different points of execution (essentially a cleanup and expansion of the already existing feature but may also be slightly different/generalized, e.g., by tracking if anything from a variant was read or only information from the base structure). This well may tie in with the above suggestion as it may allow us to systematically detect if a part of our code has used anything channel/variant specific. In such cases we may then be able to skip additional variants and re-use results (thinking about e.g. initialization of samples, I'm looking at you #319 ...)