[RTADPA BW Tests] New Sprucing bandwidth test using latest-available HLT2 input.

cc: @rjhunter.

related to: lhcb/Moore!2763 (merged) + lhcb-core/LHCbPR2HD!270 (merged) + lhcb-core/LHCbNightlyConf!1121 (merged)

Changes:

Adds a new sprucing test, Moore_spruce_latest_bandwidth, which uses the Hlt2 nominal full-stream output which was uploaded to eos on the previous day.

Added a new script, generate_hlt2_fullstream_metadata.py, which runs only for Hlt2 nominal tests and writes the metadata.yaml based on information from the full-stream output of the test currently running. This metadata.yaml is used as an input config for the new sprucing test.

• a little safety around division in zero from the calculation of overlap_matrix, never comes up in the actual tests because every stream has a non-zero retention in 100k events, but when running test jobs it can come up and it was annoying me.

• changes to facilitate the new test + metadata.

ToDo:

• Maybe revisit the N_EVTS and calculate via a gaudipython job?
  • Wait to see if BWTesters know a way to tell the N_EVTS available in a file.
  • Else, adapt list_event_numbers to extract the N_EVTS by iterating a gaudipython job.
• Rebase after !362 (merged) was merged

Closes #12 (closed)

python/MooreTests/generate_hlt2_fullstream_metadata.py

@@ -9,8 +9,9 @@
 # or submit itself to any jurisdiction.                                       #
 ###############################################################################
 '''
-    Generates a configuration yaml for the FULL stream output of an Hlt2 BW job.
-    Intended to be used as input for chained sprucing BW jobs.
+    Takes the config file describing the Hlt2 job's input file.
+    Generates a metadata yaml about the FULL stream output of the Hlt2 job.
+    This metadata can then be used as a config file for a 'latest' Spruce job's input file.
 '''
 import argparse
 import json
@@ -20,13 +21,6 @@ from PRConfig.TestFileDB import test_file_db
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument(
-        '-n',
-        '--events',
-        default=-1,
-        type=lambda x: int(round(float(x))),
-        help='nb of events to process',
-        required=True)
     parser.add_argument(
         '-c',
         '--config',
@@ -59,15 +53,15 @@ def main():
     ifile = fname_helper.event_no_fname(**fname_helper_args)
     with open(ifile, 'r') as f:
         # json = {stream: [evt_numbers]}
-        full_evts = len(json.load(f)['full'])
+        n_triggered_full = len(json.load(f)['full'])
 
-    hlt2_evts = int(parse_yaml(fname_helper.input_nevts_json())['n_evts'])
+    n_hlt2_input = int(parse_yaml(fname_helper.input_nevts_json())['n_evts'])
     input_rate = config['input_rate']
-    opts["input_rate"] = f"{input_rate * full_evts / hlt2_evts} # in kHz"
-    opts["n_evts"] = f"{full_evts}"
+    opts["input_rate"] = f"{input_rate * n_triggered_full / n_hlt2_input} # in kHz"
+    opts["n_evts"] = f"{n_triggered_full}"
 
-    ofile = fname_helper.metadata_path(**fname_helper_args)
-    with open(ofile, 'w') as f:
+    metadata_config_for_use_in_sprucing_latesthlt2_test = fname_helper.metadata_path(**fname_helper_args)
+    with open(metadata_config_for_use_in_sprucing_latesthlt2_test, 'w') as f:
         yaml.dump(opts, f, default_flow_style=False)