[RTADPA BW Tests] New Sprucing bandwidth test using latest-available HLT2 input.

cc: @rjhunter.

related to: lhcb/Moore!2763 (merged) + lhcb-core/LHCbPR2HD!270 (merged) + lhcb-core/LHCbNightlyConf!1121 (merged)

Changes:

Adds a new sprucing test, Moore_spruce_latest_bandwidth, which uses the Hlt2 nominal full-stream output which was uploaded to eos on the previous day.

Added a new script, generate_hlt2_fullstream_metadata.py, which runs only for Hlt2 nominal tests and writes the metadata.yaml based on information from the full-stream output of the test currently running. This metadata.yaml is used as an input config for the new sprucing test.

• a little safety around division in zero from the calculation of overlap_matrix, never comes up in the actual tests because every stream has a non-zero retention in 100k events, but when running test jobs it can come up and it was annoying me.

• changes to facilitate the new test + metadata.

ToDo:

• Maybe revisit the N_EVTS and calculate via a gaudipython job?
  • Wait to see if BWTesters know a way to tell the N_EVTS available in a file.
  • Else, adapt list_event_numbers to extract the N_EVTS by iterating a gaudipython job.
• Rebase after !362 (merged) was merged

Closes #12 (closed)

python/MooreTests/run_bandwidth_test_jobs.py

@@ -79,7 +79,9 @@ def run_gaudi_job(args, config, job_input):
         extra_options += ["event_store = 'EvtStoreSvc'", "use_iosvc = True"]
 
     if args.use_manifest:
-        extra_options += [f'input_manifest_file = \'{config["input_manifest_file"]}\'']
+        extra_options += [
+            f'input_manifest_file = \'{config["input_manifest_file"]}\''
+        ]
 
     extra_options = [f"options.{opt_str}" for opt_str in extra_options]
     extra_options.insert(0, "from Moore import options")
@@ -149,7 +151,8 @@ if __name__ == '__main__':
         '-um',
         '--use-manifest',
         action='store_true',
-        help="Flag to access and include config[input_manifest_file] as an extra option in the job."
+        help=
+        "Flag to access and include config[input_manifest_file] as an extra option in the job."
     )
     args = parser.parse_args()
 
@@ -167,9 +170,10 @@ if __name__ == '__main__':
 
     config = parse_yaml(args.config)
 
-
     if args.use_manifest and not "input_manifest_file" in config.keys():
-        raise KeyError(f'{args.config} does not provide "input_manifest_file" but --use-manifest is in use.')
+        raise KeyError(
+            f'{args.config} does not provide "input_manifest_file" but --use-manifest is in use.'
+        )
     if "testfiledb_key" in config.keys():
         from PRConfig.TestFileDB import test_file_db
         inputs_fns = test_file_db[config['testfiledb_key']].filenames