Add a sprucing bandwidth test

A new bandwidth tests, Moore_spruce_bandwidth, is added to monitor the rate/event size/bandwidth of Sprucing lines.

This test will be launched in LHCbPR everyday for each lhcb-master and lhcb-head build, and will replace the current Moore_spruce_rate_and_size test.

Some major changes to

• filesize.py
• line-and-stream-rates.py
• line-similarty.py
• make_bandwidth_test_page.py
• stream_overlap.py

to make the scripts also work for Sprucing test.

One big change is on the unpacking algorithm, as reading MDF is different for Sprucing.

In summary, the old unpacking algorithm

    manifest = load_tck(args.tck)
    algs = do_unpacking(manifest, input_process=args.process)
    algsnew = algs[0:4] + [algs[-1]]

is replaced by

    algs = []
    unpack = unpack_rawevent(
        bank_types=['ODIN', 'HltDecReports', 'DstData', 'HltRoutingBits'],
        configurables=True)
    hlt2 = [hlt_decisions(source="Hlt2", output_loc="/Event/Hlt2/DecReports")]
    if args.process == 'Spruce':
        spruce = [
            hlt_decisions(
                source="Spruce", output_loc="/Event/Spruce/DecReports")
        ]
    else:
        spruce = []
    decoder = decoder(input_process=args.process)
    algs = [unpack] + hlt2 + spruce + [decoder] + [createODIN(ODIN='myODIN')]

The Hlt2 results before/after this change are identical.

I've generated two preview pages for Hlt2 and Sprucing.

Requires lhcb/Moore!2365 (merged).

Work towards lhcb-dpa/project#199 and lhcb/Moore#507 (closed).

cc @enoomen @sstahl @nskidmor @abertoli

assigned to @shunan

mentioned in merge request lhcb/Moore!2365 (merged)

changed the description

Hi @sstahl , I hope we can merge this soon after !304 (merged) is done. I saw you're happy with that MR, could you please review this one? Thanks.

added 1 commit

• 730e142c - fix hlt2 test script

Compare with previous version

changed the description

added 7 commits

• 730e142c...853f86a5 - 6 commits from branch master
• 4db8e5b7 - Merge branch 'master' into add-new-bandwidth-tests

Compare with previous version

added 2 commits

• dafe634f - simplify line-and-stream-rates.py
• 7c2fbea0 - fix typo

Compare with previous version

added 1 commit

• 005b862a - Fixed formatting

Compare with previous version

approved this merge request

Hello @shunan, can you assign to somebody with karma so that this gets merged as soon as lhcb/Moore!2365 (merged) goes in? Thanks.

resolved all threads

assigned to @rmatev

merged

mentioned in commit a79bea1d

@shunan I have merged this - please keep an eye out for any problems