Bug when running athena in CI
So far we thought that the
run_athena job is connecting to lxplus and running the
run_athena.sh script over
ssh. This was actually not the case since the code will immediately exit from the ssh shell and execute
run_athena.sh locally in the gitlab runner.
There was a suggestion from Frank to try to do this over ssh however as explained below the necessary developments do not easily fit with the CI workflow, so we decided to drop it.
When running athena in the gitlab runner though we are affected by this:
09:08:44 Domain[ROOT_All] Error > Access DbDomain UPDATE Domain[ROOT_All] (UNKNOWN) impossible. [ROOT_All] 09:08:44 Domain[ROOT_All] Fatal The requested persistent backend implementation 09:08:44 Domain[ROOT_All] Fatal for the storage type:ROOT_All cannot be loaded. 09:08:44 Are you sure you loaded you loaded the correct DLLs? 09:08:44 ERROR (pool): 09:08:44 POOL> Missing driver DLL. 09:08:44 StreamEVGEN FATAL Standard std::exception is caught 09:08:44 StreamEVGEN ERROR POOL> Missing driver DLL. 09:08:44 Traceback (most recent call last): 09:08:44 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.13/InstallArea/x86_64-slc6-gcc62-opt/jobOptions/AthenaCommon/runbatch.py", line 18, in <module> 09:08:44 theApp.run() # runs until theApp.EvtMax events reached 09:08:44 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.13/InstallArea/x86_64-slc6-gcc62-opt/python/AthenaCommon/AppMgr.py", line 663, in run 09:08:44 sc = self.getHandle()._evtpro.executeRun( nEvt ) 09:08:44 Exception: StatusCode IEventProcessor::executeRun(int maxevt) => 09:08:44 POOL> Missing driver DLL. (C++ exception of type runtime_error)
which seems to indicate that something is missing on the runner.
Until we have !106 (closed) merged, we could try to see if we can either fix this or somehow handle this check without
run_athena exiting at this point.
Alternative from Frank:
singularity exec -e --no-home -B /cvmfs -B /var /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos6 /bin/bash -- setupATLASandAsetupandGentfScript.sh