AthGeneration Fixes, master branch (2020.03.20.)

These are some changes to address the issue described in ATEAM-599.

After fixing the issue with decorator not being available in the nightly's runtime environment (by updating PyUtils/CMakeLists.txt), I ran into a number of other issues as well while trying to run @ewelina's example job.

Gen_tf.py --ecmEnergy=13000 --maxEvents=10 --jobConfig=421101 --outputEVNTFile=test_bb_Jpsimu4mu4X.pool.root

• I had to add xAODEventInfoCnv (and as a consequence xAODCnvInterfaces) to AthGeneration;
  • To make the package compile successfully, I also had to update its code a bit;
• I had to fix a number of issues in skel.GENtoEVGEN.py to make it (sort of) work.

After all of this, with all these changes applied, the above mentioned job fails during configuration with:

...
generate 09:53:52 Py:Athena            INFO including file "/cvmfs/atlas.cern.ch/repo/sw/Generators/MC16JobOptions/scripts/check_jo_consistency.py"
generate 09:53:52       OK: New jobOption file
generate 09:53:52       OK: name format correct
generate 09:53:52       OK: P8B_A14NNPDF23LO_pp_JpsimumuZmumu physicsShort less than 50 characters long
generate 09:53:52       OK: 2 physicsShort parts found
generate 09:53:52       OK: Loading generator list file from cvmfs
generate 09:53:52       OK: No generator full name is found
generate 09:53:52       Generators used:  ['P8B']
generate 09:53:52 Py:Athena            INFO including file "mc.P8B_A14NNPDF23LO_pp_JpsimumuZmumu.py"
generate 09:53:52 Shortened traceback (most recent user call last):
generate 09:53:52   File "/workspaces/build/x86_64-centos7-gcc8-opt/jobOptions/EvgenJobTransforms/skel.GENtoEVGEN.py", line 245, in <module>
generate 09:53:52     include(jofile)
generate 09:53:52   File "/cvmfs/atlas.cern.ch/repo/sw/Generators/MCJobOptions/421xxx/421101/mc.P8B_A14NNPDF23LO_pp_JpsimumuZmumu.py", line 8, in <module>
generate 09:53:52     include('Pythia8B_i/Pythia8B_A14_NNPDF23LO_Common.py')
generate 09:53:52 
generate 09:53:52 IncludeError: include file Pythia8B_i/Pythia8B_A14_NNPDF23LO_Common.py can not be found
generate 09:53:52 
generate 09:53:52 Py:Athena            INFO leaving with code 2: "an error occurred when including a file"
PyJobTransforms.trfExe.execute 2020-03-20 09:53:52,938 INFO generate executor returns 2
PyJobTransforms.trfExe.validate 2020-03-20 09:53:52,939 ERROR Validation of return code failed: Non-zero return code from generate (2) (Error code 65)
PyJobTransforms.trfExe.validate 2020-03-20 09:53:52,955 INFO Scanning logfile log.generate for errors in substep None
PyJobTransforms.trfValidation.scanLogFile 2020-03-20 09:53:52,957 WARNING Detected python exception - activating python exception grabber
PyJobTransforms.transform.execute 2020-03-20 09:53:52,957 CRITICAL Transform executor raised TransformValidationException: Non-zero return code from generate (2); Logfile error in log.generate: "IncludeError: include file Pythia8B_i/Pythia8B_A14_NNPDF23LO_Common.py can not be found"
PyJobTransforms.transform.execute 2020-03-20 09:53:56,002 WARNING Transform now exiting early with exit code 65 (Non-zero return code from generate (2); Logfile error in log.generate: "IncludeError: include file Pythia8B_i/Pythia8B_A14_NNPDF23LO_Common.py can not be found")

But I decided that this I would rather just leave to the generators group to sort out...

Pinging @jchapman as well.

added bugfix label

Hi Attila the changes are included in !31325 (closed) but the CI crashes with some strange problem. Cheers, ewelinA

This merge request affects 4 packages:

• Event/xAOD/xAODEventInfoCnv
• Generators/EvgenJobTransforms
• Projects/AthGeneration
• Tools/PyUtils

Adding @rbianchi as watcher

added Build EDM Generators Tools master review-pending-level-1 labels

Hi Ewelina,

The CMake updates of xAODEventInfo/CMakeLists.txt in !31325 (closed) are not correct.

Now, the changes in xAODEventInfo are much better in this MR than in !31325 (closed). So... which MR should we designate as the "main one"? I could pull in your changes for the EvgenJobTransforms package from !31325 (closed) into this MR. Would that be okay with you?

Cheers, Attila

mentioned in merge request !31325 (closed)

Yes, it is fine. However, I have found finally the problem in Event/xAOD/xAODEventInfoCnv/CMakeLists.txt ;o) Anyway, I would really prefer that we do not pull into AthGenerators the InnerDetector/InDetConditions/InDetBeamSpotService package as we do not need it.

? This MR is not pulling that package in. And unless I missed something, it should compile fine in all projects. Actually I couldn't find where the InDetBeamSpotService dependency came from. The xAODEventInfoCnv code definitely doesn't have any compile-time dependency on that package these days. Hence the dependency is removed in this MR.

OK, thanks. So, I will close !31325 (closed).

added 1 commit

• 81bbb9e4 - Updated the jobO skeletons in EvgenJobTransforms based on !31325 (closed).

Compare with previous version

This merge request affects 4 packages:

• Event/xAOD/xAODEventInfoCnv
• Generators/EvgenJobTransforms
• Projects/AthGeneration
• Tools/PyUtils

Adding @rbianchi as watcher

added 1 commit

• 5b6fbeab - Fixed a typo, making AthGeneration CMake configurations a bit too verbose.

Compare with previous version

This merge request affects 4 packages:

• Event/xAOD/xAODEventInfoCnv
• Generators/EvgenJobTransforms
• Projects/AthGeneration
• Tools/PyUtils

Adding @rbianchi as watcher

CI Result SUCCESS (hash 81bbb9e4)

	Athena	AthSimulation	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 11231]

Thanks for this @akraszna. @fwinkl, could we possibly get this one in for the next nightly too please?

added RC Attention Required label

I believe the MR is good like this, but we should still wait for the CI results of 5b6fbeab. The success reported so far is not for the very latest version of the code.

Thanks Attila - I got excited when I saw "all green" ;-)

CI Result SUCCESS (hash 74e540ef)

	Athena	AthSimulation	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 1
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 11219]

CI Result SUCCESS (hash 5b6fbeab)

	Athena	AthSimulation	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 1
AthSimulation: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 11233]