dynamic (pre-)selection decorations for CP algorithm sequences

One problem with the configuration of CP algorithm sequences is what to do when the user wants to add or remove algorithms from the sequence. In principle that is fine, except all of our preselection decorations etc, depend on knowing the entire sequence, and they get completely thrown off if the user modifies them after the fact. So we are in the somewhat paradoxical situation in which we generate sequences that allow for various kinds of manipulations, but if the user actually tries any of them, it will completely break the sequence. :(

This MR tries to address this problem by attaching all the bookkeeping that is needed for configuring subsequent algorithms to the sequence itself. That way, the user can then rearrange and manipulate algorithms at will, and the corresponding book-keeping information gets automatically updated with it. All this meta-configuration information will be kept until the sequence gets "configured" at which point the corresponding properties will get set on the algorithms. While not 100% foolproof, it should make the whole mechanism a lot more robust and flexible. In particular it now makes it easy to add a user-defined preselection to the sequence and have it picked up automatically.

When defining CP-algorithm sequences there are two changes that need to be made:

• All the bookkeeping of cuts etc. that is currently handled in the python code defining the sequence should instead be attached as meta-configuration to the configuration of that algorithm.
• Any properties that are dependent on this bookkeeping information/meta-configuration are no longer set directly, but instead they are added as dynamic-configuration to the configuration of that algorithm.

The actual meta-configuration gets stored in the form of python dictionaries in which each entry is a python list. The main reason for the lists is that the variables this replaces always consists of lists, but it also makes it fairly easy and transparent to combine information from multiple algorithms. The reason for the dictionaries is that it affords a great amount of flexibility, while also making the dynamic configuration very clear and easy to write.

While this may sound a bit abstract, taking a look at the muon CP sequence which has been updated as part of this merge request illustrates that this is mostly a drop-in replacement of existing code. There is however concern that while in some sense the configuration becomes simpler, the actual complexity of adding a single algorithm increases.

Going through the actual changes to the muon sequence one at a time:

• In the beginning of configuration a number of variables were defined that kept track of selection decorations. These variables were removed and instead the corresponding meta-configuration variables are defined on the sequence. In practical terms this means changing

selectionDecorNames = []
selectionDecorCount = []

to

seq.addMetaConfigDefault ("selectionDecorNames", [])
seq.addMetaConfigDefault ("selectionDecorCount", [])

This mechanism can also be used by the user, e.g. to add an overall preselection to the sequence that is applied even before the first algorithm is run.

• Algorithms that add selection decorations would add the name (and number of bits) to these bookkeeping variables for subsequent use. Instead they get added to the algorithm meta-configuration instead. In practical terms this means changing

seq.append( alg, ... )
selectionDecorNames.append( alg.selectionDecoration )
selectionDecorCount.append( 2 )

to

seq.append( alg, ...,
            metaConfig = {'selectionDecorNames' : [alg.selectionDecoration],
                          'selectionDecorCount' : [2]})

• Actual users of the meta-configuration need to add a dynamic configuration for the properties that use the meta-configuration. Most typically that is the preselect property, which combines all the selection decorations from all the previous selection algorithms to define which objects are even worth looking at. The dynamic configuration usually happens via a lambda function which is given a dictionary with all the meta-configuration and then transforms the entry it needs into the property value required. While this is probably rather intimidating to new users to see, it affords a great deal of flexibility and leads to rather simple code once one is familiar with the lambda syntax. In practical terms this means changing

alg.preselection = "&&".join (selectionDecorNames)
seq.append( alg, ... )

to

seq.append( alg, ...,
            dynConfig = {'preselection' : lambda meta : "&&".join (meta["selectionDecorNames"])})

cc @akraszna @khoo @tadej @jburr @lheinric @alister