Plot/histogram-level regression testing
Suggested by @swertz : some of the open PRs show differences, so we should increase test coverage. The easiest way seems to be with a more complete analysis example that contains enough of the typical building blocks and systematic uncertainties to fill the gaps of the current tests (which are mostly unit tests for small components, and tests to check that there are no errors when using standard functions and defining plots). This requires access to some reasonable amount of data (e.g. a MC and/or data NanoAOD), so for now it won't run fully automatically on gitlab CI, but it will be integrated with pytest or a standalone script, such that it's easy to check locally (ingrid-ui1 at T2_BE_UCL, or other sites after copying a few files).
The reference output directory should also contain the config files (pointing to local paths or xrootd url's) to run the tests (one for on-demand, one for postprocessed NanoAOD), the current ones take about 10min each.
The tests can then be run with e.g. (leaving out the --plots-reference
argument disables the test; there is also a --plots-output
argument to keep the outputs instead of using a temporary directory)
pytest tests/test_plotswithreference.py --plots-reference=test_out/zmm_0204_2
Still to do:
-
postprocess the NanoAODs, such that we can validate that the two workflows do the same thing -
add more plots and/or systematics (ideas welcome) -
store reference (current master) files -
histogram comparison script (I can recycle something) -
script or pytest integration
Other ideas are welcome