Hdf5 merger
It would be quite nice to merge HDF5 files on the grid. Fortunately @jburr has written a utility to do this!
NOTE: I've kept the history from Jon's original repository in this merge request, in case it helps to merge in any final changes. I suppose we should squash this when it's ready to be merged, though.
Merge request reports
Activity
@jburr, comments welcome. If you're ok with this I'll unmark it as a WIP.
Hi @jburr, changed. Anything else you can spot?
This merge request affects 1 package:
- PhysicsAnalysis/AnalysisCommon/HDF5Utils
Adding @dguest as watcher
added 21.2 Analysis review-pending-level-1 labels
❎ CI Result FAILUREAnalysisBase AnalysisTop AthAnalysis AthDerivation externals ✅ ✅ ✅ ✅ cmake ✅ ✅ ✅ ✅ make ✅ ✅ ⭕ ✅ required tests ✅ ✅ ✅ ✅ optional tests ✅ ✅ ✅ ✅ Full details available at NICOS MR-24892-2019-07-16-01-44
✅ AnalysisBase: number of compilation errors 0, warnings 0
✅ AnalysisTop: number of compilation errors 0, warnings 0
⭕ AthAnalysis: number of compilation errors 1, warnings 257
⚠ AthDerivation: number of compilation errors 0, warnings 11
📝 For experts only: Jenkins output [CI-MERGE-REQUEST 39906]This merge request affects 1 package:
- PhysicsAnalysis/AnalysisCommon/HDF5Utils
Adding @dguest as watcher
❎ CI Result FAILUREAnalysisBase AnalysisTop AthAnalysis AthDerivation externals ✅ ✅ ✅ ✅ cmake ✅ ✅ ✅ ✅ make ✅ ✅ ⭕ ✅ required tests ✅ ✅ ✅ ✅ optional tests ✅ ✅ ✅ ✅ Full details available at NICOS MR-24892-2019-07-16-08-56
✅ AnalysisBase: number of compilation errors 0, warnings 0
✅ AnalysisTop: number of compilation errors 0, warnings 0
⭕ AthAnalysis: number of compilation errors 1, warnings 83
⚠ AthDerivation: number of compilation errors 0, warnings 1
📝 For experts only: Jenkins output [CI-MERGE-REQUEST 39908]The error also makes no sense... It appears to compile completely fine but then ends on
08:42:08 resuts_dict_generator: Info: added Build["AthDerivation"] = "SUCCESS"
08:42:08 Info: result SUCCESS, overall_result FAILURE
08:42:08 resuts_dict_generator: Info: added BuildTime["Start;AthDerivation"] = "1563258783.797408765"
08:42:08 resuts_dict_generator: Info: added BuildTime["End;AthDerivation"] = "1563259328.950837516"
08:42:08 resuts_dict_generator: Info: added BuildTime["Start"] = "1563258441.219391935"
08:42:08 resuts_dict_generator: Info: added BuildTime["End"] = "1563259328.981692101"
08:42:09 Info: recorded start build time: 1563258441.219391935, end time: 1563259328.981692101
08:42:09 Info: OVERALL RESULT: FAILURE
08:42:09 Build step 'Execute shell' marked build as failure
08:42:09 Finished: FAILURE
Edited by Jon BurrHi @dguest, @jburr, could you please add ATLAS Robot as developer to the fork? (!24892 (merged))
Thanks, Tadej (L2)
@jburr I think you're not looking at the whole log:
08:32:51 [ 32%] Building CXX object Control/AthAllocators/CMakeFiles/AthAllocators.dir/src/Arena.cxx.o 08:32:51 Scanning dependencies of target CxxUtils_make_unique_test 08:32:51 In file included from /var/lib/jenkins/workspace/CI-MERGE-REQUEST/21.2/PhysicsAnalysis/AnalysisCommon/HDF5Utils/HDF5Utils/DefaultMerger.h:8:0, 08:32:51 from /var/lib/jenkins/workspace/CI-MERGE-REQUEST/21.2/PhysicsAnalysis/AnalysisCommon/HDF5Utils/util/hdf5-merge.cxx:6: 08:32:51 /var/lib/jenkins/workspace/CI-MERGE-REQUEST/21.2/PhysicsAnalysis/AnalysisCommon/HDF5Utils/HDF5Utils/IH5Merger.h:62:15: error: 'H5::H5Location' has not been declared 08:32:51 H5::H5Location& targetLocation, 08:32:51 ^~~~~~~~~~ 08:32:51 /var/lib/jenkins/workspace/CI-MERGE-REQUEST/21.2/PhysicsAnalysis/AnalysisCommon/HDF5Utils/HDF5Utils/IH5Merger.h:71:15: error: 'H5::H5Location' has not been declared 08:32:51 H5::H5Location& targetLocation, 08:32:51 ^~~~~~~~~~ 08:32:51 In file included from /var/lib/jenkins/workspace/CI-MERGE-REQUEST/21.2/PhysicsAnalysis/AnalysisCommon/HDF5Utils/util/hdf5-merge.cxx:6:0: 08:32:51 /var/lib/jenkins/workspace/CI-MERGE-REQUEST/21.2/PhysicsAnalysis/AnalysisCommon/HDF5Utils/HDF5Utils/DefaultMerger.h:68:15: error: 'H5::H5Location' has not been declared 08:32:51 H5::H5Location& targetLocation, 08:32:51 ^~~~~~~~~~
this is likely a header mismatch since
H5::H5Location
is a "new" base class. My wild guess is that LCG is including HDF5 1.8 and cmake is building against that.This merge request affects 1 package:
- PhysicsAnalysis/AnalysisCommon/HDF5Utils
Adding @dguest as watcher