Move declaration of QMTest tests from CMake to Python
For the configuration of QMTest (.qmt
) tests we are using a mix of CMake and Python code. The CMake part finds the .qmt
files and declares (add_test
) the tests command line with some common settings (like labels), then we invoke a Python script that analyzes the dependencies of .qmt
files and generates a CMake fragment that we then include to pass the information to CMake/CTest.
Given that we have to invoke Python from CMake and that CMake language is not very practical for complex logic, we would be much better off to migrate the whole declaration of QMTest (both add_test
and dependencies) to a single Python script.
Apart from the increased flexibility, we will have other benefits:
- some IDEs (Visual Studio Code, for one) can run CTest tests and point to the place in CMake code that declare each test, which in the current case means always the same line in
GaudiToolbox.cmake
:add_test(NAME ${package_name}.${qmt_name} ...
. A Python script would generate one explicit line per test making it easier to debug test failures - having test declaration in Python we can better integrate with the test runner (
GaudiTesting.Run
) opening to the possibility of better use of CTest features like test attachments that could replace the current special test measurements implemented printing a lot of noise to stdout (see the dedicated piece of code) - (probably) generating more of the test declaration code from Python would simplify the migration from QMTest to pytest
- we could address #202 collecting test metadata for all directories and generate the test declaration code in a second step