Remove necessity to use TestFileDB in HLT2 BW tests and add new 2023 HLT2 BW test
FYI @sstahl @enoomen @shunan @lugrazet
Goes with Moore!2463, LHCbNightlyConf!1051 and lhcb-core/LHCbPR2HD!254 (merged).
The periodic BW tests are currently restricted to using inputs from the TestFileDB. We need flexibility to be able to run over samples not in the TestFileDB so we can quickly reconfigure these tests to use e.g. new samples from the HLT1 BW division team.
To achieve this, run_bandwidth_test_jobs now reads a yaml file defining the input. This can either provide a testfiledb_key (as was previously given via the cmd line) or can provide a list of input files. I chose yaml so that we can add comments and it has a nicely-formatted way to store a big list of files that can be easily read in by python. The options files live under Moore/Hlt/Hlt2Conf/tests/options/bandwdith/ and are added in Moore!2463.
I also add a new Moore_hlt2_2023_bandwidth.sh script to configure a new test on that sample. I think I've added the necessary configuration in LHCbNightlyConf!1051 and lhcb-core/LHCbPR2HD!254 (merged) for this test to run, but not sure if more changes in other projects are required.
Other small changes/small refactors
- Limited the amount of output files to be logged to 10 files. Necessary for the 2023 input as there is a huge list of input files and you don't need to print them all to the console,
- I made the downloading of input files recently added in configurable (kept it on by default) as this downloading causes a massive slowdown on my machine - perhaps the network between me and
eos/is very slow. Switched on by default still to not slow down the periodic tests. - Improve the string-formatting of the
extra-optionstogaudirun.py, - Report the correct input rate of the sample in the BW test page.
TODO:
-
Check that the HLT1 output rate is properly configured/configurable for each set of files, -
Make sure new test will run<- working locally, including downloading. -
Anything to be done toMoore_spruce_all_lines.sh? -
Address comments I've made in the code. -
Update HLT2 output rate/sprucing input rate for sprucing BW test
Merge request reports
Activity
assigned to @rjhunter
mentioned in merge request lhcb/Moore!2463 (merged)
Resolved by Ross John Hunter@rjhunter Thanks for this implementation, but this MR should have merge conflicts with !312 (merged), and I would suggest we wait until that one is done.
- Last reply by Ross John Hunter
Resolved by Ross John Hunter@rjhunter thanks for the MR, I think it is nice to have the option to include an options file which sets various options. I think for the nightly tests it would be better to keep the explicit option to run from the test file db. Could we implement it in such a way that both options are possible?
- Last reply by Ross John Hunter
added 24 commits
-
a493a7e3...e5263abb - 18 commits from branch
master - 1ba20e21 - Merge branch 'master' into rjhunter-extend-BW-testing-options
- cfa3fe5d - First attempt to parse in sample details via yaml
- d8a39d34 - Merge branch 'master' into rjhunter-extend-BW-testing-options
- 01bc09f9 - Make the ability to download files configurable and other fixes
- bd891ebb - EVTS_FIVE->EVTS_PROD (bad merge); remove unnecessary options files; symmetrise...
- 198bc1c9 - Patch to allow download or not download
Toggle commit list-
a493a7e3...e5263abb - 18 commits from branch
mentioned in merge request lhcb-core/LHCbNightlyConf!1051 (merged)
added 1 commit
- 924ed361 - Update sprucing test with new way of parsing
mentioned in merge request lhcb-core/LHCbPR2HD!254 (merged)