Skip to content

New Trigger ART test python steering framework

Rafal Bielski requested to merge rbielski/athena:trig-art-python into master

First implementation of what is described in ATR-20038, i.e. a new python framework for configuring and running Trigger ART tests. Not all features/steps are implemented yet, but it's enough to merge already and add further functionality in subsequent MRs. Five example tests are added from all four packages TriggerTest, TrigAnalysisTest, TrigUpgradeTest and TrigP1Test. They all temporarily include "_py_" in the name to avoid interference with the existing bash scripts.

The new framework is implemented as a python package TrigValSteering within TrigValTools/python. There is a README file provided.

Status of the required features described in ATR-20038:

Support different executors: athena, athenaHLT, Reco_tf, Trig_reco_tf, etc --> done with examples for the first three

Provide reasonable default configurations based on executor, data type --> provided exec step configuration based on type of executor and a default_check_steps function to create check steps list

Implement all features of the current exec scripts --> some steps/features still missing

Allow to define multiple athena (or trf) steps within one test and handle the post-exec steps for each of them and take care of merging the results --> easily possible, one example provided in TrigAnalysisTest

Allow to add custom steps --> no examples yet, but possible by using the base Step class

Ensure consistent outputs across all packages and types of tests, which integrate well with ART Monitor website generation code --> main point of the design, will be verified in the website after it runs in a nightly

By default use name of the script file to construct reference names etc, unless other name is given explicitly --> done in the RefComparisonStep base class

Check file naming convention (as much as reasonable and possible) --> done with checks on prefix and length

Return with an exit code based on sub-step results where each step can be configured to be either required or optional --> done, each step has a required parameter with some reasonable defaults (True for exec and checklog)

Facilitate running unit tests which would only verify configuration of ART tests without executing them --> to be done

Print the list of individual commands executed by the test to a separate file --> printed to commands.json

However, verification and further updates to these features will come as I implement more tests beyond the five examples. Right now, not all common check steps are implemented yet, but we have CheckLog, RegTest, HistCount, PerfMon, plus log merging, compressing and Tail.

Merge request reports