Skip to content

Option to isolate tests with a per-test working directory

Rosen Matev requested to merge rm-isolate-tests into master

Context

  1. Often race conditions between tests are introduced. Typically the problem is that multiple tests share the same options/scripts and they do not explicitly override the output filenames per test.
  2. Sometimes tests behave differently when they run for a second time (because of the presence of outputs)
  3. use_temp_dir=trueonly creates a directory per package (with a fixed path in the build directory)
  4. Some tests need the output of "prerequisite" tests.

Proposed solution

  1. Run each test in its own directory named after the test.
  2. To use this, instead of true pass per-test to the use_temp_dir argument in qmt files.
<argument name="use_temp_dir"><enumeral>per-test</enumeral></argument>
  1. If a test suite.testB depends on a file output.txt produced by test suite.testA, one can configure testB to look for
"../suite.testA/output.txt"

or the more generic (for Python gaudi options)

Path(os.getenv("PREREQUISITE_0", "")) / "output.txt"
# or the old-style
os.path.join(os.getenv("PREREQUISITE_0", ""), "output.txt")

PREREQUISITE_i environment variables are set by the test runner. The advantage w.r.t. "../suite.testA/output.txt" or os.path.expandvars("$PREREQUISITE_0/output.txt") is that running outside of the test framework would work as expected, i.e. search for output.txt in the current directory.

Open questions

  • Do not introduce per-test but change the behaviour of true?
  • Change the default behaviour of all tests and completely ignore use_temp_dir?

/cc @clemenci @cattanem

Edited by Rosen Matev

Merge request reports