Implement `GaudiExeTest` base test class and migrate qmtest tests to pytest
We introduce a pytest based alternative to QMTest test description.
Most functionalities are preserved and others are added and all tests have been converted (except for a few test of the QMTest machinery). To be noted that some tests have been renamed from the Savannah style bug id to the matching JIRA issue.
We expect tweaks and improvements later on, but the API can be considered stable enough for this to be merged.
1. Base Classes
-
SubprocessBaseTest: A base class for running and managing subprocess executions, providing the setup and utility methods for ensuring tests run in a consistent environment -
GaudiExeTest: Extends SubprocessBaseTest to handle specific LHCb workflows including additional functionalities for dealing with options, preprocessing outputs, and validating against reference files.
2. Test Configuration
- Tests are configured through class attributes specifying the command line program and arguments (
command). - The execution environment can be customized by setting the environment attribute, which accepts a list of
key=valuestrings or using theupdate_envclass method for more complex handling (helper functions likeexpand_vars_fromorunset_varsare provided). - Tests can specify a
timeoutand an expectedreturncode. - Test options are defined using the
optionsattribute: could be a method (for Python options), a string (for.opts) or a dictionary (for JSON options) - Paths for command arguments and reference files are relative to the file where the test class is defined and are automatically resolved to absolute paths using the
resolve_pathmethod.
3. Test Execution
- The
fixture_resultfixture ensures that each test class’s program executes once, using the results across all test functions. - Tests can be configured to run in a specified directory by setting
popen_kwargs["cwd"]. By default, a pytest temporary directory is used. - Additional subprocess arguments can be specified through
popen_kwargs.
4. Output Management
-
stdoutandstderrstreams are capped at 100MB. - Default preprocessing of outputs can be overridden by redefining the
preprocessorattribute inGaudiExeTest.
5. Exception Handling
- Errors such as timeouts and exceeded stream size limits are reported through
ProcessTimeoutErrorandExceededStreamErrorexceptions. - If such errors occur, the
test_fixture_setupmethod will fail after reporting all relevant information, including the failure. All other tests are skipped.
6. Test Fixtures
- Test validation times are automatically captured using the
capture_validation_timefixture. - The
capture_class_docstringfixture captures and records the docstring of a test class if it exists.
7. Standard Tests
- By default,
GaudiExeTestincludes tests for validatingstderr(checks if stderr is empty in case no error reference is provided),stdout,histosandttrees(only if a reference file is provided), andreturncode(checks if the return code 0 in case no expected value is specified). - Additional custom tests can be added using utilities like
find_reference_blockandcount_error_lines,validate_json_with_reference. - If any of
stdout,histos,ttreesorstderrcomparisons fail (the output of the program is different than the content of the reference), a.newfile is generated with the new output.
8. Platform-Specific Tests
- Tests should be skipped using the
@pytest.mark.skipifmarker and theplatform_matcheshelper function, which performs regex matching against the platform.
9. Reporting
- The
ctest_measurements_reporterplugin integrates Pytest with CTest by reporting results usingDartMeasurementXML tags. - Test configuration details are recorded using a dedicated test method,
test_fixture_setup. - Tests can report additional information through the
record_propertyfixture - The properties are prefixed with the test name and class name in the output tag to differentiate properties across tests:
ClassName.test_name.property_name
Edited by Marco Clemencic