Improve test suite
The current testing procedures are a bit fragmented, with some tests implemented using pytest
(unit tests), some in a python script (cross checking results of executables and python scripts), and some in a bash script (cross checking results of executables with reference results). A few things problematic with this:
- Only pytests run as part of CI/CD. Ideally more/all would
- One has to run everything to run all test cases, e.g. the behaviour when espilon and mu are purely real is only tested by the
.sh
script currently. - Reliance on executables and makefiles (hard to avoid completely, but as much as possible should be python)
- A bit unflexible, running the
.py
or.sh
scripts usually take a few minutes, to run a specific test one must manually comment out code. - Output of
verify_backwards_python.py
in the case oftest_empty.txt
is slightly difficult to understand: many warnings are printed - maybe these could be caught somehow; and printing in the end that there are7 errors
gives the impression the test is failing, while it is not.