[RTADPA BW Tests] New Sprucing bandwidth test using latest-available HLT2 input.

cc: @rjhunter.

related to: lhcb/Moore!2763 (merged) + lhcb-core/LHCbPR2HD!270 (merged) + lhcb-core/LHCbNightlyConf!1121 (merged)

Changes:

Adds a new sprucing test, Moore_spruce_latest_bandwidth, which uses the Hlt2 nominal full-stream output which was uploaded to eos on the previous day.

Added a new script, generate_hlt2_fullstream_metadata.py, which runs only for Hlt2 nominal tests and writes the metadata.yaml based on information from the full-stream output of the test currently running. This metadata.yaml is used as an input config for the new sprucing test.

• a little safety around division in zero from the calculation of overlap_matrix, never comes up in the actual tests because every stream has a non-zero retention in 100k events, but when running test jobs it can come up and it was annoying me.

• changes to facilitate the new test + metadata.

ToDo:

• Maybe revisit the N_EVTS and calculate via a gaudipython job?
  • Wait to see if BWTesters know a way to tell the N_EVTS available in a file.
  • Else, adapt list_event_numbers to extract the N_EVTS by iterating a gaudipython job.
• Rebase after !362 (merged) was merged

Closes #12 (closed)

assigned to @lugrazet

changed the description

added 1 commit

• 57766f58 - Functional test now, but feels a little hacky

Compare with previous version

added 1 commit

• 8614d9f6 - some minor cleanup of comments in bw_test.sh

Compare with previous version

changed the description

mentioned in merge request lhcb/Moore!2763 (merged)

changed the description

mentioned in merge request lhcb-core/LHCbPR2HD!270 (merged)

I just had a little look here @lugrazet and I think the broad outline looks good. I'll have more specific comments when you're ready for review.

We do indeed have to do something better with the N_EVTS and close #12 (closed) while we're at it. I think the analysis scripts should be working out for themselves how many events they are running over... You keep working on the new test as you are, and at some point one of us can get around to improving this N_EVTS stuff.

added 3 commits

• 8614d9f6...6f6c963b - 2 commits from branch master
• 101eed86 - Merge branch 'master' into 'lugrazet-BW-spruce-ydayshlt2'

Compare with previous version

changed the description

added 1 commit

• d3be07ce - some refactorisation of strings/names in/from bandwidth_helpers

Compare with previous version

changed the description

added 3 commits

• d3be07ce...f4c873bf - 2 commits from branch master
• 03521d8f - Merge branch 'master' into 'lugrazet-BW-spruce-ydayshlt2'

Compare with previous version

changed the description