visualize generalization & pruning
One often finds itself in a situation where one wants to run visualize repeatedly on the same sample folder. Sometimes, this is quite fast, but other times, very slow - especially if the input sample folder is not using aggressive optimization techniques. Most of the time there is then spent loading the sample folder.
In principle it would be easy to merge the sample folder a-posteriori, using the generalization techniques we employ for the merging of sample folders, but it is very cumbersome to go through all of the visualize
configs and figure out which components can be merged to what extent.
In the visualize
step, in principle, all the information required is available, such that we could implement a flag that would make the post-visualize sample folder highly optimized for future processing with the same visualize
step. All we would need to do for this is to track throughout visualize which paths are being used to what granularity, and then employ a call to generalize
at the end (and maybe prune the resulting sample folder using the same pruning techniques we already use in the analyze
step.
This would require some thought and some bookkeeping, but could potentially make the lives of analyzers a lot easier.