We introduce a pytest based alternative to QMTest test description.
Most functionalities are preserved and others are added and all tests have been converted (except for a few test of the QMTest machinery). To be noted that some tests have been renamed from the Savannah style bug id to the matching JIRA issue.
We expect tweaks and improvements later on, but the API can be considered stable enough for this to be merged.
1. Base Classes
SubprocessBaseTest
: A base class for running and managing subprocess executions, providing the setup and utility methods for ensuring tests run in a consistent environmentGaudiExeTest
: Extends SubprocessBaseTest to handle specific LHCb workflows including additional functionalities for dealing with options, preprocessing outputs, and validating against reference files.2. Test Configuration
command
).key=value
strings or using the update_env
class method for more complex handling (helper functions like expand_vars_from
or unset_vars
are provided).timeout
and an expected returncode
.options
attribute: could be a method (for Python options), a string (for .opts
) or a dictionary (for JSON options)resolve_path
method.3. Test Execution
fixture_result
fixture ensures that each test class’s program executes once, using the results across all test functions.popen_kwargs["cwd"]
. By default, a pytest temporary directory is used.popen_kwargs
.4. Output Management
stdout
and stderr
streams are capped at 100MB.preprocessor
attribute in GaudiExeTest
.5. Exception Handling
ProcessTimeoutError
and ExceededStreamError
exceptions. test_fixture_setup
method will fail after reporting all relevant information, including the failure. All other tests are skipped.6. Test Fixtures
capture_validation_time
fixture.capture_class_docstring
fixture captures and records the docstring of a test class if it exists.7. Standard Tests
GaudiExeTest
includes tests for validating stderr
(checks if stderr is empty in case no error reference is provided), stdout
, histos
and ttrees
(only if a reference file is provided), and returncode
(checks if the return code 0 in case no expected value is specified).find_reference_block
and count_error_lines
, validate_json_with_reference
.stdout
, histos
, ttrees
or stderr
comparisons fail (the output of the program is different than the content of the reference), a .new
file is generated with the new output.8. Platform-Specific Tests
@pytest.mark.skipif
marker and the platform_matches
helper function, which performs regex matching against the platform.9. Reporting
ctest_measurements_reporter
plugin integrates Pytest with CTest by reporting results using DartMeasurement
XML tags.test_fixture_setup
.record_property
fixtureClassName.test_name.property_name