Skip to content

Plot/histogram-level regression testing

Pieter David requested to merge piedavid/bamboo:addregressiontesthistos into master

Suggested by @swertz : some of the open PRs show differences, so we should increase test coverage. The easiest way seems to be with a more complete analysis example that contains enough of the typical building blocks and systematic uncertainties to fill the gaps of the current tests (which are mostly unit tests for small components, and tests to check that there are no errors when using standard functions and defining plots). This requires access to some reasonable amount of data (e.g. a MC and/or data NanoAOD), so for now it won't run fully automatically on gitlab CI, but it will be integrated with pytest or a standalone script, such that it's easy to check locally (ingrid-ui1 at T2_BE_UCL, or other sites after copying a few files).

The reference output directory should also contain the config files (pointing to local paths or xrootd url's) to run the tests (one for on-demand, one for postprocessed NanoAOD), the current ones take about 10min each. The tests can then be run with e.g. (leaving out the --plots-reference argument disables the test; there is also a --plots-output argument to keep the outputs instead of using a temporary directory)

pytest tests/test_plotswithreference.py --plots-reference=test_out/zmm_0204_2

Still to do:

  • postprocess the NanoAODs, such that we can validate that the two workflows do the same thing
  • add more plots and/or systematics (ideas welcome)
  • store reference (current master) files
  • histogram comparison script (I can recycle something)
  • script or pytest integration

Other ideas are welcome

Edited by Pieter David

Merge request reports