Skip to content

Completely rewrite test_trig_mc_v1Dev_slice_reproducibility_build

Rafal Bielski requested to merge rbielski/athena:fix-reproducibility-test into master

This test was previously relying on comparing lines from log files which matched some regex. This was broken by !30690 (merged) because the regex no longer matched. Because this test wasn't handling errors correctly, it always succeeded silently even if it couldn't match anything because of missing inputs or wrong regex. Moreover, it had some other issues in python3.

I took this opportunity to completely rewrite the test and make it based on the chainDump.py outputs in JSON format rather than on grepping logs. It still runs the same athena jobs as before, i.e. full menu and individual slices, but now the athena step saves the online monitoring histograms to a differently-named file each time. Each of the histogram outputs is then input for chainDump.py which produces a JSON file with all counts. To facilitate this, I made a minor feature extension in chainDump.py (ability to rename JSON output). This way, we exercise here the same infrastructure as everywhere else for processing trigger counts.

The comparison step now loads all the JSON data into a single dictionary and easily iterates slice by slice, chain by chain, decision by decision, and prints INFO or ERROR message saying whether the numbers match or not. I think the new format is nicer to read and easier to understand. It is now more robust against the case when one of the input files is missing because of failed previous steps. It also works in py3 as far as I could test it, so I think this fixes ATR-20996.

cc @tbold
Currently inconsistent counts discovered by the fixed test are:

ERROR HLTChain count difference HLT_xe100_mht_L1XE50, MET: 8, FullMenu: 7
ERROR HLTDecision count difference HLT_xe100_mht_L1XE50_Step0, MET: 8, FullMenu: 7
ERROR Counts for MET differ from FullMenu
ERROR HLTDecision count difference HLT_mu6noL1_L1MU6_Step1, Muon: 13, FullMenu: 14
ERROR Counts for Muon differ from FullMenu

For reviewers:
test_trig_mc_v1Dev_slice_reproducibility_build.py is completely rewritten, so it's better to just look at the new version of the file rather than the git diff.

Merge request reports