Unit, system and performance tests
This MR extends our CI to include many tests of the framework, broadly categorized in the following types:
Module Tests
Tests of individual modules. Using a fixed seed (random_seed = 0
) together with a fixed Geant4 version will always reproduce the same event. One event is produced and the DEBUG
level output of the respective module is checked using regex.
The CMakeLists.txt
automatically globs for all configurations found in the test_modules
folder and executes Allpix Squared with them. The regex to search for are also stored in the configuration file as comment (starting line with #
, no other comments in the file allowed) and will be picked up by CMake automatically.
Currently implemented tests:
- test_01_geobuilder.conf
- test_02-1_electricfield_linear.conf
- test_02-2_electricfield_init.conf
- test_03_deposition.conf
- test_04-1_propagation_project.conf
- test_04-2_propagation_generic.conf
- test_05_transfer_simple.conf
- test_06_digitization.conf
- test_07_histogramming.conf
- test_08-1_writer_root.conf
- test_09_reader_root.conf
Dependencies between test can be indicated using the #DEPENDS
keyword in the config file. See test 09 for an example, it uses the output of test 08-1 to read data back in.
Performance Tests
Similar to the implementation above, performance tests use configurations prepared such, that one particular module takes most of the load ("slowest instantiation"), and a couple of thousand events are simulated from a fixed seed. The #TIMEOUT
keyword in the configuration file will ask CTest to abort the test after the given running time.
Since fluctuations on the CI runners occur, arising from different loads of the executing machines, all performance CI jobs are marked with the allow_failure: true
keyword. This will allow GitLab to continue processing the pipeline but will mark the end-result as "passed with warnings" indicating an issue in the pipeline. This should then be checked manually.
Performance tests are now limited to "native" runners, i.e. they are not executed on docker hosts where the hypervisor decides on the number of parallel jobs. Currently this comprises only the Mac runners, but a native CentOS7 host should be up soon, tagged for performance tests.
Current tests comprise:
- test_01_deposition.conf
- test_02-1_propagation_generic.conf
- test_02-2_propagation_project.conf
- test_02-3_propagation_generic_multithread.conf
Core Tests
to be implemented