Migrating EVNTMerge_tf.py to ComponentAccumulator
In order to allow event generation-specific configuration, Generation was added to the ProductionStep Enum.
Database/IOVDbSvc/python/IOVDbAutoCfgFlags.py was then modified to set flags.IOVDb.GlobalTag='' by default in the case of a Generation job.
A new EVNTMerge_Skeleton.py module was added to EvgenJobTransforms and EVNT merging clients were updated to use this as the skeletonCA.
The non-CA support was dropped as this was broken anyway.
After this, the EVNTMerge_tf.py job from Generators/EvgenJobTransforms/test/test_full_chain_min_bias.sh runs fine.
NB One side-effect of these changes is that the Stream name in the output pool file is now StreamEVNT rather than StreamEVGEN which was used in legacy configuration jobs. The one place that this was checked (in event picking) has been updated to support both stream names.
A possible next step would be to add an EVNTMerge_tf.py CI test, this will be done in a separate MR though.
Merge request reports
Activity
Hi @jchapman for mc23 we use 23.6.3,AthGeneration Thanks, ewelinA
added 1 commit
- 893f4538 - move setting production step to GENtoEVGEN_Skeleton.py
Thanks @ewelina. Here are the diffs:
> acmd.py diff-root EVNT.ref.MERGE_pool.root EVNT2.MERGE_pool.root --error-mode resilient --mode semi-detailed xAOD::Init INFO Environment initialised for data access Py:diff-root INFO comparing tree [CollectionTree] in files: Py:diff-root INFO old: [EVNT.ref.MERGE_pool.root] Py:diff-root INFO new: [EVNT2.MERGE_pool.root] Py:diff-root INFO branches of interest: set() Py:diff-root INFO ignore leaves: ('Token', 'index_ref', '(.*)_timings\\.(.*)', '(.*)_mems\\.(.*)', '(.*)TrigCostContainer(.*)') Py:diff-root INFO enforce leaves: ('BCID',) Py:diff-root INFO leaves prefix: Py:diff-root INFO hacks: ('m_athenabarcode', 'm_token') Py:diff-root INFO entries: -1 Py:diff-root INFO mode: semi-detailed Py:diff-root INFO error mode: resilient Py:diff-root INFO order trees: False Py:diff-root INFO exact branches: False Py:diff-root INFO comparing over [100] entries... Py:diff-root INFO comparing [11] leaves over entries... Py:diff-root INFO Found [992901] identical leaves Py:diff-root INFO Found [558356] different leaves Py:diff-root INFO [McEventCollection_p6_GEN_EVENT.m_genEvents.m_e_attribute_id]: 100 leaves differ Py:diff-root INFO [McEventCollection_p6_GEN_EVENT.m_genEvents.m_e_attribute_name]: 279128 leaves differ Py:diff-root INFO [McEventCollection_p6_GEN_EVENT.m_genEvents.m_e_attribute_string]: 279128 leaves differ Py:diff-root ERROR NOTE: there were errors during the dump Py:diff-root INFO fold.allgood: False Py:diff-root INFO fnew.allgood: True Py:diff-root ERROR files differ!So here we see that the containers in CollectionTree match except for the addition/removal of some extra GenEvent level attributes (to be understood)
> acmd.py diff-pool EVNT.ref.MERGE_pool.root EVNT2.MERGE_pool.root ## opening file [EVNT.ref.MERGE_pool.root]... ## importing ROOT... ## importing ROOT... [DONE] ## opening file [OK] ## opening file [EVNT2.MERGE_pool.root]... ## importing ROOT... ## importing ROOT... [DONE] ## opening file [OK] ================================================================================ ::: Comparing POOL files... ref : EVNT.ref.MERGE_pool.root chk : EVNT2.MERGE_pool.root -------------------------------------------------------------------------------- ## ERROR: files don't have the same content !! ## collections in 'chk' and not in 'ref' + BunchId + ConditionsRun + EventNumber + EventStreamInfo_p3_StreamEVNT + EventTime + EventTimeNanoSec + EventWeight + IsCalibration + IsSimulation + IsTestBeam + LumiBlockN + McChannel + RunNumber + Token + xAOD::EventFormat_v1_EventFormatStreamEVNT + xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.amiTag ## collections in 'ref' and not in 'chk' - EventStreamInfo_p3_StreamEVGEN - xAOD::EventFormat_v1_EventFormatStreamEVGEN ================================================================================ ::: comparing common content (mem-size / disk-size)... [ERR] 1.504 / 0.579 kb (ref) ==> 1.499 / 0.614 kb (chk) | DataHeaderForm [ERR] 1.504 / 0.579 kb (ref) ==> 1.499 / 0.614 kb (chk) | DataHeaderForm [ERR] 19.896 / 1.384 kb (ref) ==> 20.874 / 3.316 kb (chk) | EventInfo_p4_McEventInfo [ERR] 1.488 / 0.630 kb (ref) ==> 1.499 / 0.684 kb (chk) | IOVMetaDataContainer_p1__TagInfo [ERR] 7743.463 / 1956.291 kb (ref) ==> 4209.185 / 1041.043 kb (chk) | McEventCollection_p6_GEN_EVENT [ERR] 1.362 / 0.268 kb (ref) ==> 2.138 / 1.621 kb (chk) | index_ref [ERR] 1.362 / 0.268 kb (ref) ==> 2.138 / 1.621 kb (chk) | index_ref [ERR] 21.024 / 0.875 kb (ref) ==> 22.099 / 3.131 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAux. [ERR] 1.136 / 0.163 kb (ref) ==> 2.421 / 1.797 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.actualInteractionsPerCrossing [ERR] 1.141 / 0.164 kb (ref) ==> 2.435 / 1.807 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.averageInteractionsPerCrossing [ERR] 1.067 / 0.153 kb (ref) ==> 2.229 / 1.660 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.mcChannelNumber [ERR] 1.456 / 0.308 kb (ref) ==> 2.601 / 2.031 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.mcEventNumber [ERR] 2.553 / 0.376 kb (ref) ==> 3.776 / 2.441 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.mcEventWeights [ERR] 2.157 / 0.359 kb (ref) ==> 3.372 / 2.393 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.streamTagDets [ERR] 2.157 / 0.359 kb (ref) ==> 3.372 / 2.393 kb (chk) | xAOD::EventAuxInfo_v3_EventInfoAuxDyn.streamTagRobs [ERR] 3.021 / 0.363 kb (ref) ==> 4.034 / 2.285 kb (chk) | xAOD::EventInfo_v1_EventInfo [ERR] 0.889 / 0.239 kb (ref) ==> 0.882 / 0.248 kb (chk) | xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAux. ================================================================================ ## Comparison : [ERR]Now comparing the size of the containers we see that the McEventCollection has reduced in size significantly (this suggests a reduction in which GenEvent attributes are persistified between 23.6.3 and the head of the 24.0 branch?) The other big difference is that the CA job wants to call the stream
StreamEVNTrather thanStreamEVGEN. Attempting to fix this causes other errors as it expects corresponding config flags with EVGEN rather than EVNT in their names. Finally, there are a few extra variables added (which diff-root ignores):+ BunchId + ConditionsRun + EventNumber + EventTime + EventTimeNanoSec + EventWeight + IsCalibration + IsSimulation + IsTestBeam + LumiBlockN + McChannel + RunNumber + Token + xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.amiTagThese mostly appear to be redundant other than the amiTag, so should probably be dropped from being written out (will follow-up).
Comparing the metadata is awkward due to the Stream name change, but otherwise things look OK:
> meta-diff.py EVNT.ref.MERGE_pool.root EVNT2.MERGE_pool.root /TagInfo has changed from 'AtlasRelease: AthGeneration-23.6.3, beam_energy: 6800000, beam_type: collisions, evgenProcess: , evgenTune: A2 MSTW2008LO, generators: Pythia8(v.309)+EvtGen(v.2.1.1), hepmc_version: HepMC3, keywords: minbias, qcd, sm, lhefGenerator: , mc_channel_number: 421113, randomSeed: 123456, specialConfiguration: NONE' to 'AMITag: e8455, AtlasRelease: Athena-24.0.41, beam_energy: 6800000, beam_type: collisions, evgenProcess: , evgenTune: A2 MSTW2008LO, generators: Pythia8(v.309)+EvtGen(v.2.1.1), hepmc_version: HepMC3, keywords: minbias, qcd, sm, lhefGenerator: , mc_channel_number: 421113, randomSeed: 123456, specialConfiguration: NONE' EventStreamInfo_p3_StreamEVGEN has been deleted EventStreamInfo_p3_StreamEVNT has been inserted FileMetaData has changed from 'beamEnergy: 6800000.0, beamType: collisions, dataType: StreamEVGEN, generatorsInfo: Pythia8(v.309)+EvtGen(v.2.1.1), lumiBlocks: [0], mcProcID: 421113.0, productionRelease: AthGeneration-23.6.3, runNumbers: [421113]' to 'amiTag: e8455, beamEnergy: 6800000.0, beamType: collisions, dataType: StreamEVNT, generatorsInfo: Pythia8(v.309)+EvtGen(v.2.1.1), lumiBlocks: [0], mcProcID: 421113.0, productionRelease: Athena-24.0.41, runNumbers: [421113]' auto_flush has changed from '-30000000' to '10' file_comp_alg has changed from '1' to '2' file_guid has changed from '25A0CA8A-0C96-5843-89C7-40621D5F9C30' to '551EBE55-C414-F84E-942C-85628050044B' file_size has changed from '2051706' to '1138953' metadata_items has changed from '/Generation/Parameters: IOVMetaDataContainer, /TagInfo: IOVMetaDataContainer, EventStreamInfo_p3_StreamEVGEN: EventStreamInfo, xAOD::EventFormat_v1_EventFormatStreamEVGEN: xAOD::EventFormat_v1, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAux.: xAOD::FileMetaDataAuxInfo_v1, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.beamEnergy: Float_t, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.beamType: string, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.generatorsInfo: string, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.mcProcID: Float_t, xAOD::FileMetaData_v1_FileMetaData: FileMetaData' to '/Generation/Parameters: IOVMetaDataContainer, /TagInfo: IOVMetaDataContainer, EventStreamInfo_p3_StreamEVNT: EventStreamInfo, xAOD::EventFormat_v1_EventFormatStreamEVNT: xAOD::EventFormat_v1, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAux.: xAOD::FileMetaDataAuxInfo_v1, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.amiTag: string, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.beamEnergy: Float_t, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.beamType: string, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.generatorsInfo: string, xAOD::FileMetaDataAuxInfo_v1_FileMetaDataAuxDyn.mcProcID: Float_t, xAOD::FileMetaData_v1_FileMetaData: FileMetaData' xAOD::EventFormat_v1_EventFormatStreamEVGEN has been deleted xAOD::EventFormat_v1_EventFormatStreamEVNT has been insertedadded 1 commit
- c75a51d1 - Force disableEventTag=True for EVNTMerge_tf.py
-
I've added a workaround to allow StreamEVGEN to be used. It probably would be nicer to clean this up though.
I suspect this change from @averbyts is the cause of the McEventCollection size reduction: !65748 (merged)
added 1 commit
- 33cbdb64 - Allow StreamEVGEN to work with flags.Output.doWriteEVNT and flags.Output.EVNTFileName
added 5 commits
- f440b2e4 - set ProductionStep.Generation in GENtoEVGEN_Skeleton.py
- 7bec7e50 - Migrate EVNTMerge_tf.py to CA
- 960a5ea4 - Force disableEventTag=True for EVNTMerge_tf.py
- de9dafe0 - OutputStreamConfig.p: Fix syntax in addToMetaData function
- cde6d994 - Allow StreamEVGEN to work with flags.Output.doWriteEVNT and flags.Output.EVNTFileName
Toggle commit listadded 6 commits
- 6294f655 - set ProductionStep.Generation in GENtoEVGEN_Skeleton.py
- 29a5cc09 - IOVDbAutoCfgFlags: Avoid setting GlobalTag for Generation jobs
- ff85893c - Let POOLtoEI_Skeleton recognise inputs containing StreamEVNT as well as StreamEVGEN as EVNT files
- e073ad0d - Migrate EVNTMerge_tf.py to CA
- 80870ef8 - Drop skeleton.EVNTMerge.py
- 89079d79 - OutputStreamConfig.py: Fix syntax in addToMetaData function
Toggle commit list- Resolved by John Derek Chapman
It all seems to be working nicely now. I've removed the broken legacy skeleton, so the --CA option isn't even required. The only wrinkle is that I see this warning every event:
StreamEVNT.Stre...WARNING Unable to retrieve AttributeList with key 'StoreGateSvc+SimpleTag'Any ideas on how to get rid of this?
- Last reply by John Derek Chapman
This merge request affects 6 packages:
- Control/AthenaConfiguration
- Database/AthenaPOOL/OutputStreamAthenaPool
- Database/EventIndex/EventIndexProducer
- Database/IOVDbSvc
- Generators/EvgenJobTransforms
- Tools/PyJobTransforms
Affected files list will not be printed in this case
Adding @ssnyder ,@fwinkl ,@mnowak ,@gemmeren ,@amete ,@maszyman as watchers
added 24.0 Core Database Generators Tools analysis-review-required full-unit-tests labels
-
Athena AthSimulation AthGeneration AnalysisBase AthAnalysis externals cmake make tests Full details available on this CI monitor view. Check the JIRA CI status board for known problems
removed analysis-review-required label
-
Two general comments:
- In long term, we should probably start writing event tag also for
EVNT. - This will conflict with !70700 (merged) which is targeting main.
- In long term, we should probably start writing event tag also for
added 1 commit
- dd1ef9a1 - Clean-up of EVNTMerge_Skeleton.py python syntax.
added 177 commits
-
dd1ef9a1...c5350577 - 169 commits from branch
atlas:24.0 - 7f14f586 - Add Generation to the ProductionStep Enum
- f1ebd150 - set ProductionStep.Generation in GENtoEVGEN_Skeleton.py
- f359b678 - IOVDbAutoCfgFlags: Avoid setting GlobalTag for Generation jobs
- d0c8253c - Let POOLtoEI_Skeleton recognise inputs containing StreamEVNT as well as StreamEVGEN as EVNT files
- d5125937 - Migrate EVNTMerge_tf.py to CA
- ef4a9c17 - Drop skeleton.EVNTMerge.py
- 3e746d74 - OutputStreamConfig.py: Fix syntax in addToMetaData function
- 4be76d8d - Clean-up of EVNTMerge_Skeleton.py python syntax.
Toggle commit list-
dd1ef9a1...c5350577 - 169 commits from branch
-
This merge request affects 7 packages:
- Control/AthenaConfiguration
- Database/AthenaPOOL/OutputStreamAthenaPool
- Database/EventIndex/EventIndexProducer
- Database/IOVDbSvc
- Event/xAOD/xAODMetaDataCnv
- Generators/EvgenJobTransforms
- Tools/PyJobTransforms
Affected files list will not be printed in this case
Adding @maszyman ,@akraszna ,@gemmeren ,@amete ,@mnowak ,@fwinkl ,@ssnyder as watchers
added EDM analysis-review-required labels
-
Athena AthSimulation AthGeneration AnalysisBase AthAnalysis externals cmake make tests Full details available on this CI monitor view. Check the JIRA CI status board for known problems
