Skip to content

Remove necessity to use TestFileDB in HLT2 BW tests and add new 2023 HLT2 BW test

Ross John Hunter requested to merge rjhunter-extend-BW-testing-options into master

FYI @sstahl @enoomen @shunan @lugrazet

Goes with Moore!2463, LHCbNightlyConf!1051 and lhcb-core/LHCbPR2HD!254 (merged).

The periodic BW tests are currently restricted to using inputs from the TestFileDB. We need flexibility to be able to run over samples not in the TestFileDB so we can quickly reconfigure these tests to use e.g. new samples from the HLT1 BW division team.

To achieve this, run_bandwidth_test_jobs now reads a yaml file defining the input. This can either provide a testfiledb_key (as was previously given via the cmd line) or can provide a list of input files. I chose yaml so that we can add comments and it has a nicely-formatted way to store a big list of files that can be easily read in by python. The options files live under Moore/Hlt/Hlt2Conf/tests/options/bandwdith/ and are added in Moore!2463.

I also add a new Moore_hlt2_2023_bandwidth.sh script to configure a new test on that sample. I think I've added the necessary configuration in LHCbNightlyConf!1051 and lhcb-core/LHCbPR2HD!254 (merged) for this test to run, but not sure if more changes in other projects are required.

Other small changes/small refactors

  • Limited the amount of output files to be logged to 10 files. Necessary for the 2023 input as there is a huge list of input files and you don't need to print them all to the console,
  • I made the downloading of input files recently added in configurable (kept it on by default) as this downloading causes a massive slowdown on my machine - perhaps the network between me and eos/ is very slow. Switched on by default still to not slow down the periodic tests.
  • Improve the string-formatting of the extra-options to gaudirun.py,
  • Report the correct input rate of the sample in the BW test page.

TODO:

  • Check that the HLT1 output rate is properly configured/configurable for each set of files,
  • Make sure new test will run <- working locally, including downloading.
  • Anything to be done to Moore_spruce_all_lines.sh?
  • Address comments I've made in the code.
  • Update HLT2 output rate/sprucing input rate for sprucing BW test
Edited by Ross John Hunter

Merge request reports