Refactor and speed-up the periodic BW tests
FYI @shunan @nskidmor @sstahl @lugrazet
Goes with lhcb/Moore!2651 (merged) and lhcb-core/LHCbPR2HD!260 (merged)
Major changes:
- Removed the jobs producing streamless MDFs. They only had 2 uses before: 1) to produce an enormous similarity matrix of all lines that no-one could ever parse, and 2) to calculate the similarities between streams. 1) is unnecessary and 2) can be achieved by comparing event numbers between different streamed MDFs.
- Big refactor of
make_bandwidth_test_page.py
to reduce code duplication and shorten. - Introduce a single wrapper
Moore_bandwidth_tests.sh
rather than having lots of code duplication across the 3Moore_{hlt2, hlt2_2023, spruce}_bandwidth.sh
wrapper scripts. The 3 wrapper scripts now just callMoore_bandwidth_tests.sh
as a subprocess with 2 arguments. - Introduce a header file of common helpers. First use case was to make a
FileNameHelper
class, which ensures we keep the number of hard-coded paths across the BW tests to a minimum. - Remove a large number of analysis jobs calculating the filesize of the MDFs. These filesizes were never used, and the GP jobs didn't cost nothing.
- Make sure the stream-config JSONs are written-out by the HLT2 options files, and those are picked up in the BW test html pages. There were a couple of scripts essentially re-working out what the stream-config was in PRConfig, which is code duplication from Moore.
- Stop hard-coding the list of expected stream names in the BW wrapper scripts. This risks getting out-of-sync with Moore, so we now read it from the output of the Moore job.
- A few plotting changes/improvements to the plots in the html page. I think they're a slight improvement, but up for debate; will show in the description.
- Make the links between pages work if building locally.
The first of these should cut the runtime of the BW tests by about a quarter. The removal of the filesize jobs might save a few minutes, in particular for the HLT2 tests.
Across-the-board there is some effort (larger in some places than others) to cleanup code duplication.
TODO:
-
Condense combine-{hlt2,spruce}-output.py
to a single script withargparse
, -
Remove hard-coding of stream names in Moore_{hlt2,spruce}_bandwidth.sh
wrapper scripts, -
Cleanup/remove other unused scripts, -
Thoroughly test all 3 BW jobs for differences that may have crept in. -
Understand slight (but not negligible) difference in similarities Jaccards. -
(more optimistic) share a single wrapper shell script, rather than 3 roughly-identical ones.
Edited by Ross John Hunter