Skip to content

Component accumulator in derivation framework - part 1: slimming, job transform, DerivationFrameworkExamples

James Catmore requested to merge jcatmore/athena:master-DAOD_CA_part1 into master

Putting into CI and review now, as agreed at software coordination on 17th March. Note this does touch code that is used for all DAOD production (specifically the SlimmingHelper=, but should have zero impact on any existing DAODs.

This is the first of several MRs to migrate the derivation framework to the component accumulator. This targets three aspects:

  • Slimming mechanism
  • Job transforms mechanism
  • Updated DerivationFrameworkExamples

Slimming

The SlimmingHelper is retained but now has two modes of operation - with and without the component accumulator.

  • In the case where the component accumulator is used, the SlimmingHelper is supplied the list of container names and types (needed to build the ItemList) in the input file via the ConfigFlag Input.TypedCollections. The ItemList is then returned by the SlimmingHelper so that it can be included in the output stream via the new CA-compliant OutputStreamConfig.py here. This has been modified to allow the setting of an AcceptAlg for filtering
  • In the second case run in the old style, the list is generated inside the SlimmingHelper from the non-CA method inputFileSummary['eventdata_items'], as before. The ItemList is directly added to the stream.
  • The existing job options still work as the method names are retained. The constructor now accepts an extra keyword argument NamesAndTypes, and has a new method to return the ItemList called GetItemList.
  • At the same time, the machinery for blocking certain containers from being added as AllVariables is removed, as it has no relevance in Run 3.

Job transforms

Basic job transforms mechanism is included:

  • transform is Derivation_tf.py with the format(s) specified with requiredDerivedFormats (equivalent to the old reductionConf
  • follows example of the OverlaySkeleton with a DerivationSkeleton
  • this skeleton works out the input/output filenames and decides which workflow to use: AOD->DAOD, EVNT->DAOD or DAOD->D2AOD
  • loops over the formats required by requiredDerivedFormats and adds the config for each format to the component accumulator
  • the configs for each format are made into python modules in python instead of job options in share

Usage:

Derivation_tf.py --CA --inputAODFile AOD.27089396._001365.pool.root.1 --outputDAODFile test.pool.root --requiredDerivedFormats TEST1 TEST3 --maxEvents 1000

DerivationFrameworkExamples

The examples are being progressively updated to use the new CA set up. The naming convention will reflect the one used in the rest of the framework, that is the config python file will have the same name as the format itself (e.g. TEST1.py, TEST3.py, etc)

Edited by James Catmore

Merge request reports