Refactor and speed-up the periodic BW tests

FYI @shunan @nskidmor @sstahl @lugrazet

Goes with lhcb/Moore!2651 (merged) and lhcb-core/LHCbPR2HD!260 (merged)

Major changes:

• Removed the jobs producing streamless MDFs. They only had 2 uses before: 1) to produce an enormous similarity matrix of all lines that no-one could ever parse, and 2) to calculate the similarities between streams. 1) is unnecessary and 2) can be achieved by comparing event numbers between different streamed MDFs.
• Big refactor of make_bandwidth_test_page.py to reduce code duplication and shorten.
• Introduce a single wrapper Moore_bandwidth_tests.sh rather than having lots of code duplication across the 3 Moore_{hlt2, hlt2_2023, spruce}_bandwidth.sh wrapper scripts. The 3 wrapper scripts now just call Moore_bandwidth_tests.sh as a subprocess with 2 arguments.
• Introduce a header file of common helpers. First use case was to make a FileNameHelper class, which ensures we keep the number of hard-coded paths across the BW tests to a minimum.
• Remove a large number of analysis jobs calculating the filesize of the MDFs. These filesizes were never used, and the GP jobs didn't cost nothing.
• Make sure the stream-config JSONs are written-out by the HLT2 options files, and those are picked up in the BW test html pages. There were a couple of scripts essentially re-working out what the stream-config was in PRConfig, which is code duplication from Moore.
• Stop hard-coding the list of expected stream names in the BW wrapper scripts. This risks getting out-of-sync with Moore, so we now read it from the output of the Moore job.
• A few plotting changes/improvements to the plots in the html page. I think they're a slight improvement, but up for debate; will show in the description.
• Make the links between pages work if building locally.

The first of these should cut the runtime of the BW tests by about a quarter. The removal of the filesize jobs might save a few minutes, in particular for the HLT2 tests.

Across-the-board there is some effort (larger in some places than others) to cleanup code duplication.

TODO:

• Condense combine-{hlt2,spruce}-output.py to a single script with argparse,
• Remove hard-coding of stream names in Moore_{hlt2,spruce}_bandwidth.sh wrapper scripts,
• Cleanup/remove other unused scripts,
• Thoroughly test all 3 BW jobs for differences that may have crept in.
• Understand slight (but not negligible) difference in similarities Jaccards.
• (more optimistic) share a single wrapper shell script, rather than 3 roughly-identical ones.

python/MooreTests/calculate_stream_overlap.py

+###############################################################################
+# (c) Copyright 2000-2023 CERN for the benefit of the LHCb Collaboration      #
+#                                                                             #
+# This software is distributed under the terms of the GNU General Public      #
+# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".   #
+#                                                                             #
+# In applying this licence, CERN does not waive the privileges and immunities #
+# granted to it by virtue of its status as an Intergovernmental Organization  #
+# or submit itself to any jurisdiction.                                       #
+###############################################################################
+''' Compare event numbers extracted from mdf outputs of different streams to
+    quantify the overlap between the streams by Jaccard similarity index.
+    Writes similarity matrix to console and out to .html for usage in BW test page.
+'''
+import argparse
+import pandas as pd
+from sklearn.metrics.pairwise import pairwise_distances
+import json
+from PRConfig.bandwidth_helpers import FileNameHelper
+
+
+def get_all_event_numbers(args):
+    fname_helper = FileNameHelper(args.process)
+    ret = {}
+    for stream in args.streams:
+        with open(
+                fname_helper.event_no_fname(args.stream_config, stream),
+                'r') as f:
+            ret.update(json.load(f))
+    return ret
+
+
+def calculate_similarity_matrix(event_numbers_by_stream):
+
+    all_event_numbers = set([
+        evt_no for evt_no_list in event_numbers_by_stream.values()
+        for evt_no in evt_no_list
+    ])
+    print(
+        f"Found {len(all_event_numbers)} unique event numbers across {len(event_numbers_by_stream.keys())} streams."
+    )
+
+    df = pd.DataFrame(
+        False,
+        index=list(all_event_numbers),
+        columns=event_numbers_by_stream.keys())
+    for stream, evt_no_list in event_numbers_by_stream.items():
+        for evt_no in evt_no_list:
+            df[stream][evt_no] = True
+
+    jaccard = 1 - pairwise_distances(
+        df.T.to_numpy(), metric='jaccard'
+    )  # .T bcuz pairwise_distance must expect the fields to take similarity between to be rows rather than columns
+    jaccard_sim_matrix_df = pd.DataFrame(
+        jaccard, columns=df.columns, index=df.columns)
+
+    return jaccard_sim_matrix_df
+
+
+def save(df, htmlpath):
+    # Generate HTML table for similarity matrix
+    html = df.to_html(float_format=lambda x: f"{x:.1%}")
+    with open(htmlpath, 'w') as f:
+        f.write(html)
+
+
+def main():
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        '-p', '--process', type=str, required=True, choices=['hlt2', 'spruce'])
+    parser.add_argument(
+        '--stream-config',
+        type=str,
+        required=True,
+        choices=["wg", "production"])
+    parser.add_argument('--streams', nargs='+', type=str, required=True)
+    args = parser.parse_args()
+    fname_helper = FileNameHelper(args.process)
+
+    event_numbers = get_all_event_numbers(args)
+    for stream in args.streams:
+        print(
+            f"Found {len(event_numbers[stream])} events for {stream} stream.")
+
+    ofile = fname_helper.jaccard_similarities_path(args.stream_config)
+    sim_matrix = calculate_similarity_matrix(event_numbers)
+    print(
+        f"Calculated similarity matrix. Printing and saving to html at {ofile}."
+    )
+    print(sim_matrix)
+    save(sim_matrix, ofile)
+
+
+if __name__ == "__main__":
+    main()