Nightly test to monitor max candidates WARNINGs
Following #737 myself and @rjhunter discussed implementing a nightly test that will report what combiners hit the maximum candidates limit. The requirements are as follows
- A nightly test similar to throughput/BW tests that runs over ~1M NoBias events from 2024
- The jobs fails if a NEW combiner hits this WARNING (a simple counter?)
- A handler that extracts the combiners hitting the limit from the log and reports this back to a mattermost channel where all liaisons are present
This will obviously be for HLT2 lines initially - where the urgency really lies. A spruce version could follow however given many combiners are shared this should already be improved by work on HLT2.
The task should be fairly manageable and we could use it as an opportunity to train a new person in these types of test