Option to isolate tests with a per-test working directory
Context
- Often race conditions between tests are introduced. Typically the problem is that multiple tests share the same options/scripts and they do not explicitly override the output filenames per test.
- Sometimes tests behave differently when they run for a second time (because of the presence of outputs)
-
use_temp_dir=true
only creates a directory per package (with a fixed path in the build directory) - Some tests need the output of "prerequisite" tests.
Proposed solution
- Run each test in its own directory named after the test.
- To use this, instead of
true
passper-test
to theuse_temp_dir
argument in qmt files.
<argument name="use_temp_dir"><enumeral>per-test</enumeral></argument>
- If a test
suite.testB
depends on a fileoutput.txt
produced by testsuite.testA
, one can configure testB to look for
"../suite.testA/output.txt"
or the more generic (for Python gaudi options)
Path(os.getenv("PREREQUISITE_0", "")) / "output.txt"
# or the old-style
os.path.join(os.getenv("PREREQUISITE_0", ""), "output.txt")
PREREQUISITE_i
environment variables are set by the test runner. The advantage w.r.t. "../suite.testA/output.txt"
or os.path.expandvars("$PREREQUISITE_0/output.txt")
is that running outside of the test framework would work as expected, i.e. search for output.txt
in the current directory.
Open questions
-
Do not introduce per-test
but change the behaviour oftrue
? -
Change the default behaviour of all tests and completely ignore use_temp_dir
?
Edited by Rosen Matev