Skip to content

Implement `GaudiExeTest` base test class and migrate qmtest tests to pytest

Silia Taider requested to merge staider/Gaudi:GaudiExeTest into master

We introduce a pytest based alternative to QMTest test description.

Most functionalities are preserved and others are added and all tests have been converted (except for a few test of the QMTest machinery). To be noted that some tests have been renamed from the Savannah style bug id to the matching JIRA issue.

We expect tweaks and improvements later on, but the API can be considered stable enough for this to be merged.

1. Base Classes

  • SubprocessBaseTest: A base class for running and managing subprocess executions, providing the setup and utility methods for ensuring tests run in a consistent environment
  • GaudiExeTest: Extends SubprocessBaseTest to handle specific LHCb workflows including additional functionalities for dealing with options, preprocessing outputs, and validating against reference files.

2. Test Configuration

  • Tests are configured through class attributes specifying the command line program and arguments (command).
  • The execution environment can be customized by setting the environment attribute, which accepts a list of key=value strings or using the update_env class method for more complex handling (helper functions like expand_vars_from or unset_vars are provided).
  • Tests can specify a timeout and an expected returncode.
  • Test options are defined using the options attribute: could be a method (for Python options), a string (for .opts) or a dictionary (for JSON options)
  • Paths for command arguments and reference files are relative to the file where the test class is defined and are automatically resolved to absolute paths using the resolve_path method.

3. Test Execution

  • The fixture_result fixture ensures that each test class’s program executes once, using the results across all test functions.
  • Tests can be configured to run in a specified directory by setting popen_kwargs["cwd"]. By default, a pytest temporary directory is used.
  • Additional subprocess arguments can be specified through popen_kwargs.

4. Output Management

  • stdout and stderr streams are capped at 100MB.
  • Default preprocessing of outputs can be overridden by redefining the preprocessor attribute in GaudiExeTest.

5. Exception Handling

  • Errors such as timeouts and exceeded stream size limits are reported through ProcessTimeoutError and ExceededStreamError exceptions. 
  • If such errors occur, the test_fixture_setup method will fail after reporting all relevant information, including the failure. All other tests are skipped.

6. Test Fixtures

  • Test validation times are automatically captured using the capture_validation_time fixture.
  • The capture_class_docstring fixture captures and records the docstring of a test class if it exists.

7. Standard Tests

  • By default, GaudiExeTest includes tests for validating stderr (checks if stderr is empty in case no error reference is provided), stdout , histos and ttrees (only if a reference file is provided), and returncode (checks if the return code 0 in case no expected value is specified).
  • Additional custom tests can be added using utilities like find_reference_block and count_error_lines, validate_json_with_reference.
  • If any of stdout, histos, ttrees or stderr comparisons fail (the output of the program is different than the content of the reference), a .new file is generated with the new output.

8. Platform-Specific Tests

  • Tests should be skipped using the @pytest.mark.skipif marker and the platform_matches helper function, which performs regex matching against the platform.

9. Reporting

  • The ctest_measurements_reporter plugin integrates Pytest with CTest by reporting results using DartMeasurement XML tags.
  • Test configuration details are recorded using a dedicated test method, test_fixture_setup.
  • Tests can report additional information through the record_property fixture
  • The properties are prefixed with the test name and class name in the output tag to differentiate properties across tests: ClassName.test_name.property_name
Edited by Marco Clemencic

Merge request reports