Allow per-substep threading overrides in transforms

If we want to run RDOtoRDOTrigger for reprocessing using 21.0 and MP, but the rest in 22.0 and MT, we need to be able to override threading flags per-substep.

This then allows to run like

Reco_tf.py --multithreaded ... --asetup "RDOtoRDOTrigger:Athena,21.0,latest"

In this case everything will run in MT, but RDOtoRDOTrigger will run in MP as a legacy release is detected.

/cc @tsulaia @rbielski @okumura @jpanduro @miochoa @jferrand

Maybe @emoyse and @elmsheus would also be interested.

For the large scale AthenaMT MC tests with pathena this works right now as follows:

pathena --trf "ATHENA_PROC_NUMBER=0 Reco_tf.py --athenaopts 'h2r:--threads=8' 'RDOtoRDOTrigger:--nprocs=0' 'r2e:--threads=8' 'e2a:--threads=8' .... --asetup 'r2t:Athena,21.0.119' ...

added 1 commit

• 4cbe175f - Detect running legacy releases (21.X) in transforms and force MP for those

Compare with previous version

I went a step forward and add a (hardcoded unfortunately) check for running 21.X releases in master transforms. In such cases MT will fall-back to MP for those. We can also discuss it next week at the SW Coordination during my transforms talk.

This allows now just

Reco_tf.py --multithreaded ... --asetup "RDOtoRDOTrigger:Athena,21.0,latest"

Should we push this one in or are there better ideas how to detect if running a substep with 21.0?

added 1663 commits

• 4cbe175f...357b89a5 - 1660 commits from branch atlas:master
• 5f16afff - Allow per-substep threading overrides in transforms
• 6d45d2b3 - Detect running legacy releases (21.X) in transforms and force MP for those
• 0e7a4bcf - Improve asetup version parsing in transforms

Compare with previous version

marked this merge request as ready

changed the description

WARNING: big files (>100K) are found in the changeset

104K in file Tools/PyJobTransforms/python/trfExe.py

This merge request affects 1 package:

• Tools/PyJobTransforms

This merge request affects 5 files:

• Tools/PyJobTransforms/python/trfExe.py
• Tools/PyJobTransforms/python/trfMPTools.py
• Tools/PyJobTransforms/python/trfMTTools.py
• Tools/PyJobTransforms/python/trfUtils.py
• Tools/PyJobTransforms/test/test_trfUtils.py

added Tools master review-pending-level-1 labels

CI Result SUCCESS (hash 0e7a4bcf)

	Athena	AthSimulation	AthGeneration	AnalysisBase	AthAnalysis
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
AthAnalysis: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 28863]

The code changes look fine. Please resolve the threads if all the discussion is finished, we can then approve. Pienpen(L1)

added review-user-action-required label and removed review-pending-level-1 label

resolved all threads

added review-pending-level-1 label and removed review-user-action-required label

added review-approved label and removed review-pending-level-1 label

merged

mentioned in commit 9d8e6332