Implement `GaudiExeTest` base test class and migrate qmtest tests to pytest
We introduce a pytest based alternative to QMTest test description.
Most functionalities are preserved and others are added and all tests have been converted (except for a few test of the QMTest machinery). To be noted that some tests have been renamed from the Savannah style bug id to the matching JIRA issue.
We expect tweaks and improvements later on, but the API can be considered stable enough for this to be merged.
1. Base Classes
-
SubprocessBaseTest
: A base class for running and managing subprocess executions, providing the setup and utility methods for ensuring tests run in a consistent environment -
GaudiExeTest
: Extends SubprocessBaseTest to handle specific LHCb workflows including additional functionalities for dealing with options, preprocessing outputs, and validating against reference files.
2. Test Configuration
- Tests are configured through class attributes specifying the command line program and arguments (
command
). - The execution environment can be customized by setting the environment attribute, which accepts a list of
key=value
strings or using theupdate_env
class method for more complex handling (helper functions likeexpand_vars_from
orunset_vars
are provided). - Tests can specify a
timeout
and an expectedreturncode
. - Test options are defined using the
options
attribute: could be a method (for Python options), a string (for.opts
) or a dictionary (for JSON options) - Paths for command arguments and reference files are relative to the file where the test class is defined and are automatically resolved to absolute paths using the
resolve_path
method.
3. Test Execution
- The
fixture_result
fixture ensures that each test class’s program executes once, using the results across all test functions. - Tests can be configured to run in a specified directory by setting
popen_kwargs["cwd"]
. By default, a pytest temporary directory is used. - Additional subprocess arguments can be specified through
popen_kwargs
.
4. Output Management
-
stdout
andstderr
streams are capped at 100MB. - Default preprocessing of outputs can be overridden by redefining the
preprocessor
attribute inGaudiExeTest
.
5. Exception Handling
- Errors such as timeouts and exceeded stream size limits are reported through
ProcessTimeoutError
andExceededStreamError
exceptions. - If such errors occur, the
test_fixture_setup
method will fail after reporting all relevant information, including the failure. All other tests are skipped.
6. Test Fixtures
- Test validation times are automatically captured using the
capture_validation_time
fixture. - The
capture_class_docstring
fixture captures and records the docstring of a test class if it exists.
7. Standard Tests
- By default,
GaudiExeTest
includes tests for validatingstderr
(checks if stderr is empty in case no error reference is provided),stdout
,histos
andttrees
(only if a reference file is provided), andreturncode
(checks if the return code 0 in case no expected value is specified). - Additional custom tests can be added using utilities like
find_reference_block
andcount_error_lines
,validate_json_with_reference
. - If any of
stdout
,histos
,ttrees
orstderr
comparisons fail (the output of the program is different than the content of the reference), a.new
file is generated with the new output.
8. Platform-Specific Tests
- Tests should be skipped using the
@pytest.mark.skipif
marker and theplatform_matches
helper function, which performs regex matching against the platform.
9. Reporting
- The
ctest_measurements_reporter
plugin integrates Pytest with CTest by reporting results usingDartMeasurement
XML tags. - Test configuration details are recorded using a dedicated test method,
test_fixture_setup
. - Tests can report additional information through the
record_property
fixture - The properties are prefixed with the test name and class name in the output tag to differentiate properties across tests:
ClassName.test_name.property_name
Edited by Marco Clemencic