Bulk compare
Description
In #127 we've discussed some fully automated way of knowing when there's differences between branches. Instead, this MR just adds the ability to run a local check in an easy and repeatable way. It introduces two new scripts -
- compare-test-dumper-outputs.py - this takes two directories, which themselves contain test dumper outputs, and then compares the files (much easier now that we have h5py!)
- compare-with-main.sh - this takes as input a working branch, as well as an optional second 'base' branch (which defaults to main). This checks out the base branch, builds with both athanalysis and athena, and then does the same for the working branch. It then runs every test we have in the dumper CI, for both builds for both branches. It then uses the new compare-test-dumper-outputs.py script to ensure no unexpected changes slip through in a MR.
This allows us to make large changes, for example !930 (merged), and be certain we haven't broken any outputs for anyone.
I'm not sure if we could run this automatically. For now I'm expecting it more as a manual final check before we merge.
Review checklist:
-
CI Passing -
Comments addressed -
Source branch is up to date with target
Edited by Nikita Ivvan Pond