Run3 File Summary Record

This MR introduces a new File Summary Record implementation for Run3.

A few key aspects of the implementation are:

• leverage on the new Gaudi::Monitoring infrastructure
• simple (JSON) persistent format recorded in the ROOT file as a simple string

These features guarantee that:

• it is trivial to add a new entry to the FSR
• extracting the FSR from a ROOT file is trivial:

  import json
  import ROOT
  f = ROOT.TFile.Open("my_file.root")
  fsr = json.loads(str(f.FileSummaryRecord))

At the moment there is still much work to do (reading, merging, more frameworks info to record, ...), but it's a reasonable proof of concept.

Requires gaudi/Gaudi!1379 (merged) gaudi/Gaudi!1389 (merged) !3830 (merged)

Note: that !3830 (merged) is needed only because without it LHCb does not compile if gaudi/Gaudi!1389 (merged) is applied

References:

• https://its.cern.ch/jira/browse/LBCOMP-58
• https://indico.cern.ch/event/1086171/?note=176068
• https://indico.cern.ch/event/1110473/

added 1 commit

• 529dd091 - Initial prototype for Run3 File Summary Record

Compare with previous version

added 1 commit

• 0a4f0451 - Initial prototype for Run3 File Summary Record

Compare with previous version

added RTA label

added 1 commit

• 79f0aae1 - Initial prototype for Run3 File Summary Record

Compare with previous version

added 2 commits

• b43f4e07 - Allow for multiple FSRs in a ROOT file
• ba4ccf75 - Add service to gather stats on opened files for the FSR

Compare with previous version

added 1 commit

• d3b0785f - Add option to periodically write FSR::Sink OutputFile

Compare with previous version

added 19 commits

• d3b0785f...011f2eb5 - 14 commits from branch master
• 07b57cfe - Initial prototype for Run3 File Summary Record
• 6a8989ac - Allow for multiple FSRs in a ROOT file
• eda7af1e - Add service to gather stats on opened files for the FSR
• 40092f4f - Add option to periodically write FSR::Sink OutputFile
• dbe66033 - Improve FileEvents reports in FSR

Compare with previous version

added 1 commit

• b0dea8e3 - Optionally add the job options to the FSR

Compare with previous version

added 5 commits

• b6439837 - Add service to gather stats on opened files for the FSR
• bf7c0099 - Add option to periodically write FSR::Sink OutputFile
• d297d135 - Improve FileEvents reports in FSR
• 5800b6dd - Optionally add the job options to the FSR
• a6040073 - Improve OutputFileStats report in FSR

Compare with previous version

added 1 commit

• 64495c35 - Add support for FSR with multiple output streams

Compare with previous version

added 21 commits

• 64495c35...1c493810 - 13 commits from branch master
• 7ac2ce5c - Initial prototype for Run3 File Summary Record
• 1b5798b6 - Allow for multiple FSRs in a ROOT file
• 5f69f538 - Add service to gather stats on opened files for the FSR
• 10e2319c - Add option to periodically write FSR::Sink OutputFile
• 6ecf2835 - Improve FileEvents reports in FSR
• abe9dae7 - Optionally add the job options to the FSR
• 91eb3b25 - Improve OutputFileStats report in FSR
• a1026a16 - Add support for FSR with multiple output streams

Compare with previous version

added 57 commits

• a1026a16...f534b5d9 - 48 commits from branch master
• a0991ae8 - Initial prototype for Run3 File Summary Record
• 6dbe6a9b - Allow for multiple FSRs in a ROOT file
• 12fd6f8e - Add service to gather stats on opened files for the FSR
• b84c1321 - Add option to periodically write FSR::Sink OutputFile
• ea75e447 - Improve FileEvents reports in FSR
• 3e86539f - Optionally add the job options to the FSR
• cb18134b - Improve OutputFileStats report in FSR
• 704164ce - Add support for FSR with multiple output streams
• e3547f90 - Include FSRs from input files to output FSR

Compare with previous version

changed the description

added needs Gaudi release label

added 10 commits

• c36d2b40 - Initial prototype for Run3 File Summary Record
• 45310098 - Allow for multiple FSRs in a ROOT file
• 1262d2c7 - Add service to gather stats on opened files for the FSR
• 2ccef1fd - Add option to periodically write FSR::Sink OutputFile
• 24ea98ec - Improve FileEvents reports in FSR
• 19b6d9cd - Optionally add the job options to the FSR
• 2db7ba75 - Improve OutputFileStats report in FSR
• c44c1f83 - Add support for FSR with multiple output streams
• e9c18551 - Include FSRs from input files to output FSR
• e70faec6 - Merge FSR data from input files into counters with the same id

Compare with previous version

changed the description

added 125 commits

• e70faec6...820bf659 - 115 commits from branch master
• f825f3e9 - Initial prototype for Run3 File Summary Record
• 552bc808 - Allow for multiple FSRs in a ROOT file
• 5ebc960b - Add service to gather stats on opened files for the FSR
• cdf1cb35 - Add option to periodically write FSR::Sink OutputFile
• 8b95a993 - Improve FileEvents reports in FSR
• ef466258 - Optionally add the job options to the FSR
• 602aad16 - Improve OutputFileStats report in FSR
• f05e38d4 - Add support for FSR with multiple output streams
• cce409e3 - Include FSRs from input files to output FSR
• 28610e6b - Merge FSR data from input files into counters with the same id

Compare with previous version

added 28 commits

• 28610e6b...e4b85cc2 - 18 commits from branch master
• f9bfede4 - Initial prototype for Run3 File Summary Record
• 7bfacad5 - Allow for multiple FSRs in a ROOT file
• 3c21991a - Add service to gather stats on opened files for the FSR
• 830071f4 - Add option to periodically write FSR::Sink OutputFile
• fbd2bfe7 - Improve FileEvents reports in FSR
• c378bc63 - Optionally add the job options to the FSR
• 46c768cd - Improve OutputFileStats report in FSR
• 88c67ae5 - Add support for FSR with multiple output streams
• 67f902c9 - Include FSRs from input files to output FSR
• 31539320 - Merge FSR data from input files into counters with the same id

Compare with previous version

Good morning @clemenci. Eventually DPA will need to have some FSR information passed on, to be tested as part of our commissioning. This MR is a major step forward for using FSRs. Could you tell us the status of this and whether a version 0 could be undrafted and put in soon? I do not know how this ties with the Lumi group work, though.

CC @balagura, @egraveri, @nskidmor, @pkoppenb.