Skip to content

Refactor and speed-up the periodic BW tests

Ross John Hunter requested to merge rjhunter-trim-fat-from-BW-tests into master

FYI @shunan @nskidmor @sstahl @lugrazet

Goes with lhcb/Moore!2651 (merged) and lhcb-core/LHCbPR2HD!260 (merged)

Major changes:

  • Removed the jobs producing streamless MDFs. They only had 2 uses before: 1) to produce an enormous similarity matrix of all lines that no-one could ever parse, and 2) to calculate the similarities between streams. 1) is unnecessary and 2) can be achieved by comparing event numbers between different streamed MDFs.
  • Big refactor of make_bandwidth_test_page.py to reduce code duplication and shorten.
  • Introduce a single wrapper Moore_bandwidth_tests.sh rather than having lots of code duplication across the 3 Moore_{hlt2, hlt2_2023, spruce}_bandwidth.sh wrapper scripts. The 3 wrapper scripts now just call Moore_bandwidth_tests.sh as a subprocess with 2 arguments.
  • Introduce a header file of common helpers. First use case was to make a FileNameHelper class, which ensures we keep the number of hard-coded paths across the BW tests to a minimum.
  • Remove a large number of analysis jobs calculating the filesize of the MDFs. These filesizes were never used, and the GP jobs didn't cost nothing.
  • Make sure the stream-config JSONs are written-out by the HLT2 options files, and those are picked up in the BW test html pages. There were a couple of scripts essentially re-working out what the stream-config was in PRConfig, which is code duplication from Moore.
  • Stop hard-coding the list of expected stream names in the BW wrapper scripts. This risks getting out-of-sync with Moore, so we now read it from the output of the Moore job.
  • A few plotting changes/improvements to the plots in the html page. I think they're a slight improvement, but up for debate; will show in the description.
  • Make the links between pages work if building locally.

The first of these should cut the runtime of the BW tests by about a quarter. The removal of the filesize jobs might save a few minutes, in particular for the HLT2 tests.

Across-the-board there is some effort (larger in some places than others) to cleanup code duplication.

TODO:

  • Condense combine-{hlt2,spruce}-output.py to a single script with argparse,
  • Remove hard-coding of stream names in Moore_{hlt2,spruce}_bandwidth.sh wrapper scripts,
  • Cleanup/remove other unused scripts,
  • Thoroughly test all 3 BW jobs for differences that may have crept in.
  • Understand slight (but not negligible) difference in similarities Jaccards.
  • (more optimistic) share a single wrapper shell script, rather than 3 roughly-identical ones.
Edited by Ross John Hunter

Merge request reports