Optimization of the Trigger menu generation in CA

One more step of optimization of the menu generation with CA. The menu sequence creation is postponed during the Dataflow creation only when a new Step is found, so that their CAs are build only once. To allow this, the strategy is as follow:

• in createDataflow() the step is stored in a CFGroup object, which also accumulates the information of all the chains that use that step, and build the CA only the first time it is found in the loop of chains (this happens in the CFGroup constructor). Any new chain that uses the same step just updates the chain information in the CFgroup (using addStepLeg()). This requires some changes in the order in which the dataflow connections and the hypo tools are made. There are possible optimisations in merging some of the classes (for example CFSequence and CFGroup), but this will happen in a second step. In this MR I tried to separate the responsibilities of the different classes
• in addChainsToDataFlow() hypo tools of all the CFGroups are created
• createControlFlow() remains unchanged
• the ChainConfigs produced by the signatures are now “empty", because they contain ChanSteps that do not have CAs inside, just functions. For this reason they cannot be used in a later step, where the CFGroups are used instead (this is the reason of the change in getChainsForPrefetching )
• This latter behaviour can be changed by replacing the steps in the ChainConfigs with reference to the corresponding step object on the CFGroup, which contains the sequences. But this is another possible optimization (not clear if this makes things more complex), also related to other changes in the ChainStep class for example.

Detailed changes:

• Trigger/TrigAlgorithms/TrigGenericAlgs/python/TrigGenericAlgsConfig.py: changed due to the use of the CFSequence list instead of the Chain list. No change of behaviour
• Trigger/TriggerCommon/TriggerMenuMT/python/HLT/Config/ControlFlow/HLTCFComponents.py:
  • removal of aliases in the CFSequence class and change of name Filter to FilterNode;
  • new class CFGroup to collect Filter+Step + list of chains/dictionaries linked to the step;
  • for each chain, CFGroup.addStepLeg() adds the chain legs to the list;
  • move some functionalities from CFSequence to CFGroup (for example createHypoTools and the connection between the filter and the step)
• Trigger/TriggerCommon/TriggerMenuMT/python/HLT/Config/ControlFlow/HLTCFConfig.py:
  • remove obsolete createStepRecoNode and createCFTree;
  • new function addChainsToDataFlow() to handle the setting of the chains into the CF algorithms (no change of behaviour, only made it modular); for this reason createDataFlow() does not need dictionaries anymore;
  • in createDataFlow() the sequence CAs are created (chainStep.createSequences()) differently for the ‘fast’ and ‘slow’ menu generation: for each chain (slow) and only for the first chain (fast); no need to flag CA as merged (wasMerged()) for the fast generation
  • decisionTreeFromChains() now returns the CFGroup list (CF_list) which is used in other places;
  • createControlFlow() now uses CFGroups instead of CFSequences
• Trigger/TriggerCommon/TriggerMenuMT/python/HLT/Config/ControlFlow/HLTCFDot.py: changes needed because or removal of aliases in the component classes
• Trigger/TriggerCommon/TriggerMenuMT/python/HLT/Config/Utility/ChainMerging.py: remove obsolete noPrecedingStepsPreMerge and noPrecedingStepsPostMerge;
• Trigger/TriggerCommon/TriggerMenuMT/python/HLT/Config/GenerateMenuMT.py: makeHLTTree now returning the CF_list to be passed to prefetchingInitialRoIConfig();
• Trigger/TriggerCommon/TriggerMenuMT/python/HLT/Config/MenuComponents.py: clean-up of multiple createHypoTools() functions

With these changes:

Fast:menu generation of Dev_pp_run3_v1
real	5m14.445s
user	4m59.170s
sys	0m7.416s

Slow menu generation of Dev_pp_run3_v1

real	11m5.134s
user	10m41.054s
sys	0m14.160s

The confToo.py --diff of the HLTJobptions.json shows only two differences in the order of the chains included in the Prefetching, which is unavoidable because they are set based on a different filling. As far as I know, this is not critical in the algorithm, so I would accept it, but need experts to confirm ( @tamartin please can you confirm). Please see file attached diff