[RTADPA BW Tests] New Sprucing bandwidth test using latest-available HLT2 input.

cc: @rjhunter.

related to: lhcb/Moore!2763 (merged) + lhcb-core/LHCbPR2HD!270 (merged) + lhcb-core/LHCbNightlyConf!1121 (merged)

Changes:

Adds a new sprucing test, Moore_spruce_latest_bandwidth, which uses the Hlt2 nominal full-stream output which was uploaded to eos on the previous day.

Added a new script, generate_hlt2_fullstream_metadata.py, which runs only for Hlt2 nominal tests and writes the metadata.yaml based on information from the full-stream output of the test currently running. This metadata.yaml is used as an input config for the new sprucing test.

• a little safety around division in zero from the calculation of overlap_matrix, never comes up in the actual tests because every stream has a non-zero retention in 100k events, but when running test jobs it can come up and it was annoying me.

• changes to facilitate the new test + metadata.

ToDo:

• Maybe revisit the N_EVTS and calculate via a gaudipython job?
  • Wait to see if BWTesters know a way to tell the N_EVTS available in a file.
  • Else, adapt list_event_numbers to extract the N_EVTS by iterating a gaudipython job.
• Rebase after !362 (merged) was merged

Closes #12 (closed)

python/MooreTests/run_bandwidth_test_jobs.py

@@ -49,10 +49,20 @@ def is_remote(url):
     return url.startswith('mdf:root:') or url.startswith('root:')
 
 
+def dump_nevts(n_evts, process):
+    fname_helper = FileNameHelper(process)
+    ofile = fname_helper.input_nevts_json()
+    with open(ofile, 'w') as f:
+        json.dump({"n_evts": f"{n_evts}"}, f)
+    return 0
+
+
 def run_gaudi_job(args, config, job_input):
     # Build command line
     n_evts = input_nevts(
-        input_files=job_input, process=args.process, evtmax=args.evt_max)
+        input_files=job_input, evtmax=args.evt_max)
+    dump_nevts(n_evts, args.process)
+
     extra_options = [
         f"n_threads = {args.threads}", f"n_event_slots = {args.evtSlots}",
         f"evt_max = {n_evts}",