Bug when running athena in CI
So far we thought that the run_athena job is connecting to lxplus and running the run_athena.sh script over ssh. This was actually not the case since the code will immediately exit from the ssh shell and execute run_athena.sh locally in the gitlab runner.
There was a suggestion from Frank to try to do this over ssh however as explained below the necessary developments do not easily fit with the CI workflow, so we decided to drop it.
When running athena in the gitlab runner though we are affected by this:
09:08:44 Domain[ROOT_All] Error > Access DbDomain UPDATE Domain[ROOT_All] (UNKNOWN) impossible. [ROOT_All]
09:08:44 Domain[ROOT_All] Fatal The requested persistent backend implementation
09:08:44 Domain[ROOT_All] Fatal for the storage type:ROOT_All cannot be loaded.
09:08:44 Are you sure you loaded you loaded the correct DLLs?
09:08:44 ERROR (pool):
09:08:44 POOL> Missing driver DLL.
09:08:44 StreamEVGEN FATAL Standard std::exception is caught
09:08:44 StreamEVGEN ERROR POOL> Missing driver DLL.
09:08:44 Traceback (most recent call last):
09:08:44 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.13/InstallArea/x86_64-slc6-gcc62-opt/jobOptions/AthenaCommon/runbatch.py", line 18, in <module>
09:08:44 theApp.run() # runs until theApp.EvtMax events reached
09:08:44 File "/cvmfs/atlas.cern.ch/repo/sw/software/21.6/AthGeneration/21.6.13/InstallArea/x86_64-slc6-gcc62-opt/python/AthenaCommon/AppMgr.py", line 663, in run
09:08:44 sc = self.getHandle()._evtpro.executeRun( nEvt )
09:08:44 Exception: StatusCode IEventProcessor::executeRun(int maxevt) =>
09:08:44 POOL> Missing driver DLL. (C++ exception of type runtime_error)which seems to indicate that something is missing on the runner.
Until we have !106 (closed) merged, we could try to see if we can either fix this or somehow handle this check without run_athena exiting at this point.
Alternative from Frank:
singularity exec -e --no-home -B /cvmfs -B /var /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos6 /bin/bash -- setupATLASandAsetupandGentfScript.sh