Minor update to BW test page

Work towards lhcb-dpa/project#268. Link to lhcb-core/LHCbPR2HD!264 (merged). The updated webpages can be previewed here:

• hlt2 example
• spruce example

Addressing the following comments from @erodrigu :

Can we get a header with the slot and date, which identifies the thing, so say "Slot lhcb-head, 2023-08-15" or similar?

Now the start and end times are posted to the page so that one could know how long does the test take.

The really useful table contains lines such as

| 6.717 | 11.75475 | 38.215658 | 0.449216 | 32.157306 | 0.378001 |
|-------|----------|-----------|----------|-----------|----------|

We should really round those numbers to get something meaningful ;-).

Now all numbers in tables are rounded with 3 significant digits (some are 2 because the last digit is zero).

The table gives bandwidth in GB/s (good) but then the plots down in the page rather use MB/s. Better use GB/s everywhere (also in what gets posted via the GitLab bot, BTW).

Now use GB/s everywhere, also for the Gitlab feedback of the Sprucing throughput test. The x-axis of the BW plots now ranges in (-4. 0) (BW numbers are 0.x in terms of GB/s).

Use "s" for the unit of wall time, not "Sec". Again for consistency.

Done

Some other comments are already addressed before, some are not addressed yet and I put them in the description of the issue.

cc @rjhunter @lugrazet @nskidmor @abertoli

python/MooreTests/calculate_stream_overlap.py

@@ -30,7 +30,7 @@ def get_all_event_numbers(args):
     return ret
 
 
-def calculate_similarity_matrix(event_numbers_by_stream):
+def get_event_number_matrix(event_numbers_by_stream):
 
     all_event_numbers = set([
         evt_no for evt_no_list in event_numbers_by_stream.values()
@@ -48,6 +48,11 @@ def calculate_similarity_matrix(event_numbers_by_stream):
         for evt_no in evt_no_list:
             df[stream][evt_no] = True
 
+    return df
+
+
+def calculate_similarity_matrix(df):
+
     jaccard = 1 - pairwise_distances(
         df.T.to_numpy(), metric='jaccard'
     )  # .T bcuz pairwise_distance must expect the fields to take similarity between to be rows rather than columns
@@ -57,6 +62,18 @@ def calculate_similarity_matrix(event_numbers_by_stream):
     return jaccard_sim_matrix_df
 
 
+def calculate_overlap_matrix(df):
+    cond_prob_per_stream = {stream: [] for stream in df.columns}
+    for target_stream in df.columns:
+        for comparison_stream in df.columns:
+            cond_prob_per_stream[target_stream].append(
+                sum(df[comparison_stream] * df[target_stream]) / sum(
+                    df[target_stream]))
+    overlap_matrix_df = pd.DataFrame(
+        cond_prob_per_stream, columns=df.columns, index=df.columns)
+    return overlap_matrix_df
+
+
 def save(df, htmlpath):
     # Generate HTML table for similarity matrix
     html = df.to_html(float_format=lambda x: f"{x:.1%}")
@@ -83,14 +100,23 @@ def main():
         print(
             f"Found {len(event_numbers[stream])} events for {stream} stream.")
 
+    df = get_event_number_matrix(event_numbers)
+
     ofile = fname_helper.jaccard_similarities_path(args.stream_config)
-    sim_matrix = calculate_similarity_matrix(event_numbers)
+    sim_matrix = calculate_similarity_matrix(df)
     print(
         f"Calculated similarity matrix. Printing and saving to html at {ofile}."
     )
     print(sim_matrix)
     save(sim_matrix, ofile)
 
+    ofile = fname_helper.overlap_matrix_path(args.stream_config)
+    overlap_matrix = calculate_overlap_matrix(df)
+    print(
+        f"Calculated overlap matrix. Printing and saving to html at {ofile}.")
+    print(overlap_matrix)
+    save(overlap_matrix, ofile)
+
 
 if __name__ == "__main__":
     main()