ATR-28398: Implement possibility for a transform substep to run within a container with the specified OS

For tests running simulation of MC for Run 2, after the migration of CI and nightly builds to el9, the RDOtoRDOTRigger step is not working anymore, since it is running within the release build 21.0,Athena,latest, which does not exist in el9.
This MR fixes the issue:

1. if a release older than Rel. 24 is detected for a substep running on a non-centos7 machine, a centos7 container is invoked by default for this substep via e.g./cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/container/startContainer.sh -c centos7 -r ./runwrapper.RDOtoRDOTrigger.sh instead of the usual ./runwrapper.RDOtoRDOTrigger.sh ;
2. in addition, a flag --runInContainer is added for the Reco_tf.py transform, which allows overriding this behaviour and to specify any container for any substep. Whenever a run in a container is required, the --asetup option must also be provided.

In case the step is performed within a container, the wrapper sh script contains some info lines for a developer on how to run the wrapper for local debugging.

There is one subtlety: the container functionality requires the script $(ATLAS_LOCAL_ROOT_BASE)/container/startContainer.sh to exist (i.e. to not be renamed and to have the execute bit on). The normal way to invoke a container for a user in cmd setupATLAS -c centos7, which expands to source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh cannot be used in the transform executor, because the latter usess subprocess.Popen() with Shell=False, and this combination does not allow sourcing scripts, while the script itself is not executable. Changing to Shell=True would be a risk.

Tested successfuly on an el9 machine using commands like:

ATHENA_CORE_NUMBER=0 Reco_tf.py  --AMIConfig q443 --imf False --CA all:True RDOtoRDOTrigger:False --conditionsTag default:OFLCOND-MC16-SDR-RUN2-11 RDOtoRDOTrigger:OFLCOND-MC16-SDR-RUN2-08-02 --maxEvents 1

and

ATHENA_CORE_NUMBER=0 Reco_tf.py  --AMIConfig q443 --runInContainer RDOtoRDOTrigger:centos7 --imf False --CA all:True RDOtoRDOTrigger:False --conditionsTag default:OFLCOND-MC16-SDR-RUN2-11 RDOtoRDOTrigger:OFLCOND-MC16-SDR-RUN2-08-02 --maxEvents 1

Putting the MR as a draft for the time being. Pinging @jchapman, @nstyles, @fwinkl, @jcatmore, @elmsheus, @tadej for any comments.

AtlasTest/CITest/Athena.cmake

@@ -60,9 +60,8 @@ atlas_add_citest( OverlayRun3MC_Legacy
 atlas_add_citest( RecoRun2Data
    SCRIPT RunWorkflowTests_Run2.py --CI -r -w DataReco -e '--CA True --maxEvents 25 --conditionsTag CONDBR2-BLKPA-RUN2-11 --preExec pass' )
 
-# Temporarily disabled for el9 migration: ATR-28398
-#atlas_add_citest( RecoRun2MC
-#   SCRIPT RunWorkflowTests_Run2.py --CI -r -w MCReco --threads 0 -e '--CA "all:True" "RDOtoRDOTrigger:False" --conditionsTag "default:OFLCOND-MC16-SDR-RUN2-11" "RDOtoRDOTrigger:OFLCOND-MC16-SDR-RUN2-08-02" --maxEvents 25' )
+atlas_add_citest( RecoRun2MC
+   SCRIPT RunWorkflowTests_Run2.py --CI -r -w MCReco --threads 0 -e '--CA "all:True" "RDOtoRDOTrigger:False" --conditionsTag "default:OFLCOND-MC16-SDR-RUN2-11" "RDOtoRDOTrigger:OFLCOND-MC16-SDR-RUN2-08-02" --maxEvents 25' )
 
 atlas_add_citest( RecoRun2MC_PileUp
    SCRIPT RunWorkflowTests_Run2.py --CI -p -w MCPileUpReco -e '--CA True --maxEvents 5 --inputRDO_BKGFile=../../PileUpPresamplingRun2/run_d1730/myRDO.pool.root --conditionsTag OFLCOND-MC16-SDR-RUN2-11 --preExec="flags.Exec.FPE=500;" ' --no-output-checks  # go two levels up as the test runs in a subfolder