New python tool to create a sorted event collection

This is event sorter as requested for RDO premixing - to read events in the LumiBlock order

To use it in a transform, add --poestExec step like this:

from CollectionUtilities.SortedCollectionCreator import *
sorter = SortedCollectionCreator(name="SortEvents")
sorter.execute(runArgs.inputRDOFile, outputCollection="sortedRDORefs", sortAttribute="LumiBlockN")

ServiceMgr.EventSelector.InputCollections = ["sortedRDORefs.root"]
ServiceMgr.EventSelector.CollectionType = "ExplicitROOT"

Adding @gemmeren and @amete for discussion

This merge request affects 1 package:

• Database/APR/CollectionUtilities

This merge request affects 3 files:

• Database/APR/CollectionUtilities/CMakeLists.txt
• Database/APR/CollectionUtilities/python/SortedCollectionCreator.py
• Database/APR/CollectionUtilities/python/init.py

Adding @mnowak ,@ssnyder as watchers

added Database master review-pending-level-1 labels

CI Result SUCCESS (hash 804ac32d)

	Athena	AthSimulation	AthGeneration	AnalysisBase	AthAnalysis	DetCommon
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthAnalysis: number of compilation errors 0, warnings 0
DetCommon: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 37072]

Hi @mnowak, would this create a sorted copy of the output or does it work in the memory?

added Digitization label

Thanks for this @mnowak!

This tool creates a sorted list (aka collection) of input events - first in memory (for sorting) and then writes it into a file so it can be passed to the actual Athena job step instead of the original inputs. If this is used on the merging step, we basically do no extra work (the input files are only scanned - the event data is not read).

The 5 lines I listed need to be included at the end - that allows the transform script to configure everything using the original inputs, but at the last moment we switch the input to the sorted collection instead.

I am still thinking what is the best way to reduce these 5 lines to something even simpler, so the postExec is not so cluttered. I am thinking about creating a small script fragment with these 5 lines and use postInclude instead. Any ideas from your side? And if we create such a small script, what would be the best place to put it in?

Cheers, Marcin

At the end this will be toggled by a flag in a production job so it will be just a standard configuration fragment.

For Run 3 we will most likely also move to ComponentAccumulator-based configuration so this workflow will also have to be supported there.

Sounds good to me!
But does it mean you wish to take over the integration part from here?
Cheers, Marcin

What I meant is you should prepare a job options (and ideally also CA-based configuration fragment) that can be easily pulled-in into existing jobs.

added 1 commit

• 99cb6bff - Replaces type()== with isinstance()

Compare with previous version

This merge request affects 1 package:

• Database/APR/CollectionUtilities

This merge request affects 3 files:

• Database/APR/CollectionUtilities/CMakeLists.txt
• Database/APR/CollectionUtilities/python/SortedCollectionCreator.py
• Database/APR/CollectionUtilities/python/init.py

Adding @mnowak ,@ssnyder as watchers

resolved all threads

Hi @tadej - I think a joboptions fragment is what I was considering to be used for a postInclude. Where would such think best go to? (I don't think CollectionUtilities is a good place for it). RecJobTransforsm/share maybe?

Cheers, Marcin

It should actually go directly where the code is, why decouple it? This will be used in simulation and/or digitization jobs so RecJobTransforms is not the best place.

Let's merge this one as it is and leave the integration for a follow-up MR.

Yes, I don't intend to put it in this MR.
But I was thinking RecJobTransforms is a good place, because the Merge skeleton is there, and these joboptions basically extend the merging step (at least for the first use).
CollectionUtilies is definitely not the right place for transform joboptions - it's a very low level package.
Any other possible location candidates? :)

CI Result SUCCESS (hash 99cb6bff)

	Athena	AthSimulation	AthGeneration	AnalysisBase	AthAnalysis	DetCommon
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 1
AthSimulation: number of compilation errors 0, warnings 1
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthAnalysis: number of compilation errors 0, warnings 0
DetCommon: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 37108]

The code looks good to me. If I understand the discussion above, any changes will go in another MR so we can approve this one now. If I got that wrong @tadej or @mnowak should shout now!

-- L1