Add a sprucing bandwidth test

A new bandwidth tests, Moore_spruce_bandwidth, is added to monitor the rate/event size/bandwidth of Sprucing lines.

This test will be launched in LHCbPR everyday for each lhcb-master and lhcb-head build, and will replace the current Moore_spruce_rate_and_size test.

Some major changes to

• filesize.py
• line-and-stream-rates.py
• line-similarty.py
• make_bandwidth_test_page.py
• stream_overlap.py

to make the scripts also work for Sprucing test.

One big change is on the unpacking algorithm, as reading MDF is different for Sprucing.

In summary, the old unpacking algorithm

    manifest = load_tck(args.tck)
    algs = do_unpacking(manifest, input_process=args.process)
    algsnew = algs[0:4] + [algs[-1]]

is replaced by

    algs = []
    unpack = unpack_rawevent(
        bank_types=['ODIN', 'HltDecReports', 'DstData', 'HltRoutingBits'],
        configurables=True)
    hlt2 = [hlt_decisions(source="Hlt2", output_loc="/Event/Hlt2/DecReports")]
    if args.process == 'Spruce':
        spruce = [
            hlt_decisions(
                source="Spruce", output_loc="/Event/Spruce/DecReports")
        ]
    else:
        spruce = []
    decoder = decoder(input_process=args.process)
    algs = [unpack] + hlt2 + spruce + [decoder] + [createODIN(ODIN='myODIN')]

The Hlt2 results before/after this change are identical.

I've generated two preview pages for Hlt2 and Sprucing.

Requires lhcb/Moore!2365 (merged).

Work towards lhcb-dpa/project#199 and lhcb/Moore#507 (closed).

cc @enoomen @sstahl @nskidmor @abertoli

python/MooreTests/combine-spruce-rate-output.py

+###############################################################################
+# (c) Copyright 2000-2022 CERN for the benefit of the LHCb Collaboration      #
+#                                                                             #
+# This software is distributed under the terms of the GNU General Public      #
+# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".   #
+#                                                                             #
+# In applying this licence, CERN does not waive the privileges and immunities #
+# granted to it by virtue of its status as an Intergovernmental Organization  #
+# or submit itself to any jurisdiction.                                       #
+###############################################################################
+'''
+    Each method works the same: reads in all relevant CSV files and combines into single dataframe
+    Writes out CSV and HTML for each df
+
+'''
+import pandas as pd
+
+
+def rates_all_lines():
+
+    streams = [
+        'b_to_open_charm', 'rd', 'bandq', 'qee', 'charm', 'b_to_charmonia',
+        'slepton', 'c_to_dimuon', 'bnoc', 'dimuon'
+    ]
+    frames = []
+
+    for stream in streams:
+        file = f'tmp/Output/Inter/rates-all-lines-wg-stream-{stream}.csv'
+        try:
+            df = pd.read_csv(file, header=None)  # no lines, empty file
+        except:
+            continue
+        frames.append(df)
+
+    df = pd.concat(frames)
+    df.columns = [
+        'Line', 'Total Retention (%)', 'Rate (kHz)', 'Exclusive Retention(%)',
+        'Exclusive Rate (kHz)', 'Avg Total Event Size (kB)',
+        'Total Bandwidth (GB/s)', 'Avg DstData Size (kB)',
+        'DstData Bandwidth (GB/s)'
+    ]
+
+    df = df.sort_values(
+        by=['Total Retention (%)'], ascending=False).reset_index(drop=True)
+    df.to_csv('tmp/Output/rates-for-all-lines.csv')
+
+    html = df.to_html()
+    with open('tmp/Output/rates-for-all-lines.html', 'w') as f:
+        f.write(html)
+
+    return
+
+
+def rates_wg_streams():
+
+    streams = [
+        'b_to_open_charm', 'rd', 'bandq', 'qee', 'charm', 'b_to_charmonia',
+        'slepton', 'c_to_dimuon', 'bnoc', 'dimuon'
+    ]
+    frames = []
+
+    for stream in streams:
+        file = f'tmp/Output/Inter/rates-per-stream-wg-stream-{stream}.csv'
+        df = pd.read_csv(file, header=None)
+        frames.append(df)
+
+    df = pd.concat(frames)
+    df.columns = [
+        'Stream', 'Total Retention (%)', 'Rate (kHz)',
+        'Avg Total Event Size (kB)', 'Total Bandwidth (GB/s)',
+        'Avg DstData Size (kB)', 'DstData Bandwidth (GB/s)'
+    ]
+
+    df = df.sort_values(
+        by=['Total Retention (%)'], ascending=False).reset_index(drop=True)
+    df.to_csv('tmp/Output/rates-wg-stream-configuration.csv')
+
+    html = df.to_html()
+    with open('tmp/Output/rates-wg-stream-configuration.html', 'w') as f:
+        f.write(html)
+
+    return
+
+
+rates_all_lines()
+rates_wg_streams()