Skip to content

Use CTest fixtures and --repeat

Rosen Matev requested to merge rmatev/Gaudi:use-ctest-fixtures into master

Use FIXTURES_REQUIRED for QMTest test dependencies

For a nice introduction to FIXTURES_* in CTest see here.

FIXTURES_REQUIRED is set in addition to setting DEPENDS for ever test that defines prerequisites. Using FIXTURES_SETUP a fixture is defined for every test that is a dependency of another, where the fixture name is identical to the test name.

The main advantage of using fixtures is usability:

  1. ctest -R <regex> will also run the dependencies, unless explicitly ignored (e.g. with -FA .*).
  2. Moreover, when a dependency (fixture) fails, the dependent test is skipped ("Not Run").

CI: Use ctest --repeat instead of retrying jobs

With ctest --repeat until-pass:3, tests are run until successful or at most 3 times. Tests are retried on any failure (timeout or other).

NB: ideally we should only retry after timeouts with ctest --repeat after-timeout:3. The problem is that timeouts are handled by our test wrapper and from the CTest point of view they look like ordinary failures. There is a workaround we can do to achieve capturing stack traces and get the Timeout status but the caveat is that execution time will be reported as 0. Cleaner solutions (and the workaround) are discussed at https://gitlab.kitware.com/cmake/cmake/-/issues/17288.

Miscellaneous

  • Fix edge case in QMTest discovery (gaudi.qms/qmt_args.qmt was wrongly contracted to gaudi_args)
  • CI: Do not use deprecated globally-defined variables
  • Looser conversion from QMTest to CTest test name
  • Fail when a QMTest prerequisite test does not exist
Edited by Rosen Matev

Merge request reports