[RTADPA BW Tests] New Sprucing bandwidth test using latest-available HLT2 input.

cc: @rjhunter.

related to: lhcb/Moore!2763 (merged) + lhcb-core/LHCbPR2HD!270 (merged) + lhcb-core/LHCbNightlyConf!1121 (merged)

Changes:

Adds a new sprucing test, Moore_spruce_latest_bandwidth, which uses the Hlt2 nominal full-stream output which was uploaded to eos on the previous day.

Added a new script, generate_hlt2_fullstream_metadata.py, which runs only for Hlt2 nominal tests and writes the metadata.yaml based on information from the full-stream output of the test currently running. This metadata.yaml is used as an input config for the new sprucing test.

• a little safety around division in zero from the calculation of overlap_matrix, never comes up in the actual tests because every stream has a non-zero retention in 100k events, but when running test jobs it can come up and it was annoying me.

• changes to facilitate the new test + metadata.

ToDo:

• Maybe revisit the N_EVTS and calculate via a gaudipython job?
  • Wait to see if BWTesters know a way to tell the N_EVTS available in a file.
  • Else, adapt list_event_numbers to extract the N_EVTS by iterating a gaudipython job.
• Rebase after !362 (merged) was merged

Closes #12 (closed)

python/MooreTests/run_bandwidth_test_jobs.py

@@ -78,6 +78,9 @@ def run_gaudi_job(args, config, job_input):
     if args.download_input_files:
         extra_options += ["event_store = 'EvtStoreSvc'", "use_iosvc = True"]
 
+    if args.use_manifest:
+        extra_options += [f"input_manifest_file = {config['input_manifest_file']}"]
+
     extra_options = [f"options.{opt_str}" for opt_str in extra_options]
     extra_options.insert(0, "from Moore import options")
 
@@ -142,6 +145,12 @@ if __name__ == '__main__':
         help='Comma separated paths to directories, one per job, where the '
         'input files will be cached (default is hostname dependent or '
         '$XDG_RUNTIME_DIR).')
+    parser.add_argument(
+        '-um',
+        '--use-manifest',
+        action='store_true',
+        help="Flag to access and include config[input_manifest_file] as an extra option in the job.
+    )
     args = parser.parse_args()
 
     logging.basicConfig(
@@ -158,6 +167,9 @@ if __name__ == '__main__':
 
     config = parse_yaml(args.config)
 
+
+    if args.use_manifest and not "input_manifest_file" in config.keys():
+        raise KeyError(f'{args.config} does not provide "input_manifest_file" but --use-manifest is in use.')
     if "testfiledb_key" in config.keys():
         from PRConfig.TestFileDB import test_file_db
         inputs_fns = test_file_db[config['testfiledb_key']].filenames