Externals and Analysis Updates, master branch (2020.09.25.)

This is a pretty loaded merge request...

It updates all projects to atlasexternals-2.0.79. The list of changes wrt. atlasexternals-2.0.78 are the following (atlasexternals@2.0.78...2.0.79):

• Updated FindSYCL.cmake for better compatibility with the latest oneAPI versions;
• Updated FindUUID.cmake to set up a correct RPM dependency in case the LCG release provides this package (which it has not done for a while now);
  • Updated the build of yampl to pick up libuuid from the LCG release, if it is provided by the LCG release (should not affect the build in master);
• Updated the build of HDF5 to work correctly using Xcode 12.0 on macOS Catalina;
• Updated the build of ROOT to be more robust in using the Python version built as part of the same project as it is (should not affect the x86_64-centos7-gcc8-opt builds).

On top of these I also decided to make a few updates in this repository:

• After the failed attempt in !36360 (merged), I'm trying to clean up the build configuration of Control/RootUtils once more. Let's see what the CI will have to say about it. (The update is necessary to be able to build AnalysisBase on Ubuntu and macOS platforms.)
• After the discussion in !36641 (merged), I decided to fix the test in PhotonVertexSelection myself. Fixing ATLASG-1553.
  • I also added the package to the Athena project after fixing its test for that project as well.
• Fixed the standalone tests of TriggerAnalysisAlgorithms. Which were tweaked, but not actually fixed, in !36646 (merged).

The small change in AthenaPoolCnvSvc was needed to make the test in PhotonVertexSelection work. But since that was an actual shortcoming in that package, I decided to just fix it outright instead of working around it in the test.

Pinging @kpachal, @krumnack.

This merge request affects 10 packages:

• Control/RootUtils
• Database/AthenaPOOL/AthenaPoolCnvSvc
• PhysicsAnalysis/Algorithms/TriggerAnalysisAlgorithms
• PhysicsAnalysis/ElectronPhotonID/PhotonVertexSelection
• Projects/AnalysisBase
• Projects/AthDataQuality
• Projects/AthGeneration
• Projects/AthSimulation
• Projects/Athena
• Projects/VP1Light

Adding @jchapman ,@akraszna ,@rbianchi ,@krumnack ,@vpascuzz ,@tadej ,@ssnyder ,@pagessin ,@mnowak as watchers

added Analysis Build Core Database Egamma Trigger master review-pending-level-1 labels

@krumnack, note that while I was able to build AnalysisBase on macOS Catalina (with Xcode 12.0) with all these updates, testing-wise it still behaves terribly. (70 tests failed out of ~250.)

One of the issues I managed to identify is this one: https://github.com/root-project/root/issues/6454 (I may need to ping the ROOT developers on these things later on. I opened another issue yesterday, but nobody reacted to it yet. It seems to me that the migration from CERN's JIRA to GitHub's issue tracker has still some kinks to be worked out...)

This is just to temper your expectations with how easy it will be to use macOS as your main development platform. (You should really not...)

CI Result FAILURE (hash 96f219cb)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 21004]

Just so there wouldn't be any misunderstanding: The MR can not be accepted in its current form. It once again breaks certain Athena jobs because of the changes in how libRootUtils.so is being linked.

I'll be looking at that issue more today...

added 127 commits

• 96f219cb...20e9b098 - 124 commits from branch atlas:master
• e4915fe3 - Merge remote-tracking branch 'upstream/master' into AnalysisUpdates-master-20200925
• 4a12bdfb - Simplified the RootUtils build configuration to the absolute minimum.
• beeb65b3 - Made RootUtils.PyROOTFixes force-load GaudiKernel.

Compare with previous version

This merge request affects 10 packages:

• Control/RootUtils
• Database/AthenaPOOL/AthenaPoolCnvSvc
• PhysicsAnalysis/Algorithms/TriggerAnalysisAlgorithms
• PhysicsAnalysis/ElectronPhotonID/PhotonVertexSelection
• Projects/AnalysisBase
• Projects/AthDataQuality
• Projects/AthGeneration
• Projects/AthSimulation
• Projects/Athena
• Projects/VP1Light

Adding @jchapman ,@akraszna ,@rbianchi ,@krumnack ,@vpascuzz ,@tadej ,@ssnyder ,@pagessin ,@mnowak as watchers

hi @akraszna , I pointed some changes to code that wasn't modified in this MR but that are relevant. Everything is very well documented, that's very much appreciated.

Cheers, Santiago (L1)

added review-user-action-required label and removed review-pending-level-1 label

added review-pending-level-2 label and removed review-user-action-required label

CI Result FAILURE (hash beeb65b3)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 21084]

Crud... Since we don't use that python module anywhere in the standalone code, I thought I could get away with loading libGaudiKernel unconditionally. I just forgot that I enabled the unit test of RootUtils for the standalone build as well...

So it will have to wait for one more fix after all...

added review-user-action-required label and removed review-pending-level-2 label

added 1 commit

• e937e8cf - Made RootUtils.PyROOTFixes force-load GaudiKernel.

Compare with previous version

This merge request affects 10 packages:

• Control/RootUtils
• Database/AthenaPOOL/AthenaPoolCnvSvc
• PhysicsAnalysis/Algorithms/TriggerAnalysisAlgorithms
• PhysicsAnalysis/ElectronPhotonID/PhotonVertexSelection
• Projects/AnalysisBase
• Projects/AthDataQuality
• Projects/AthGeneration
• Projects/AthSimulation
• Projects/Athena
• Projects/VP1Light

Adding @jchapman ,@akraszna ,@rbianchi ,@krumnack ,@vpascuzz ,@tadej ,@ssnyder ,@pagessin ,@mnowak as watchers

added review-pending-level-1 label and removed review-user-action-required label

Note that at first I was trying to protect the cppyy.load_library(...) call with an exception handling block. As that would've made the hack in PyROOTFixes.py more independent of how cppyy works internally. But doing so would've resulted in an elaborate error message when loading this module in AnalysisBase. So in the end I decided for this implementation.

if cppyy.gbl.gSystem.FindDynamicLibrary( 'libGaudiKernel', True ):
    cppyy.load_library( 'libGaudiKernel' )
    pass

I'm not super happy about it, as the UI of cppyy and ROOT mix in a really muddy way here, but this is the best that I could come up with... (There's just no function in cppyy's UI for checking if a library is available or not. Internally cppyy itself also relies of TSystem to perform such checks.)

mentioned in merge request atlasexternals!739 (merged)

CI Result SUCCESS (hash e937e8cf)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 21147]

added review-approved urgent labels

removed review-pending-level-1 label

I know that I'm not supposed to do this, but I'd really like this to be included into tonight's nightly. (We have a plan to have AnalysisBase-22.2.0 installed tomorrow.)

So... could the current release coordinator give a blessing to this? (If there's no reaction, I'll just merge it myself eventually. The AnalysisBase nightlies start building after 3AM, so we still have some time.)

resolved all threads

There is a trigger hackathon this week so if you are confident it will not be too disruptive go ahead and merge.

One can never be absolutely sure, but I'm confident enough...

merged

mentioned in commit 227cbfce

added sweep:ignore label

Unfortunately this update is the one that tainted some build nodes. I should have labeled this MR as full-build. (That by itself would've helped.) I just really thought that this one time the externals update would not require it...

In case anybody is keeping count:

• This MR performed 3 builds. Tainting 2 CI nodes in the process. (2 of the 3 builds ran on the same node, aibuild20-044. Which is quite curious btw. I don't know why the second job itself didn't fail. Do we still have a cleanup during the week?)
• I counted 5 jobs that failed because of it so far, which had to receive the full-build label.

Unfortunately though there's no guarantee that any of those 5 jobs will clean up any of the 2 tainted CI nodes. @aundrus, I think we can't avoid doing some manual cleanup. The problematic machines are: aibuild20-044 and aibuild16-020. Could you clean their "master build directories"?

mentioned in merge request atlasexternals!744 (merged)

mentioned in merge request !36693 (merged)

Unfortunately I reached this notification quite late today because if overwhelming meetings number :-( . The aibuild20-044 and aibuild16-020 are cleaned already by the jobs with the full-build label set.

But yes, there is no any prediction on which machine the particular MR job will run. Manual cleanups is the only reliable way to fix 'tainted' local build areas.

mentioned in merge request !43107 (merged)