Skip to content

Optional LHCbPR tests with ci-test

Motivation

Some LHCbPR tests are too expensive to run and/or don't make sense to be run on every ci-test. Therefore, it would be good to be able to define "optional" LHCbPR tests and have a way to run them as needed.

User interface

  • An optional test does not run by default.
  • Labeling any MR included in a build will enable running an optional test. The label is defined in the test schedule xml.
  • Labeling should ideally work for non ci-test builds as well.
  • For /ci-test, reference slots (lhcb-master or lhcb-master-ref with /ci-test --build-reference ...) can not be affected by labels (since a build could be a reference to many lhcb-master-mr builds). If an automatic comparison is necessary, the test schedule should be such that the tests always run for the reference slots.

Implementation

Declaring optional tests

  <periodictest>
    <schedule type="week" time="13:45">Mon,Tue,Wed,Thu,Fri,Sat,Sun</schedule>
    <slot>lhcb-master-mr</slot>
+   <label>PR/hlt2_rate_and_size|PR/rate tests<label>
    <project>Moore</project>
    <platform>x86_64_v3-centos7-gcc11-opt+g</platform>
    <test runner="lhcbpr" group="Moore_hlt2_rate_and_size" env="lb-run-force-platform|RateTestHandler"/>
    <os_label>nightly-tests-j16</os_label>
    <count>1</count>
  </periodictest>
  • <label> is a tag similar to <slot> that is regex that should match any MR label. In the example above, having a label PR/rate tests would trigger the optional test.
  • Can the <label> tag be optional as to avoid adding <label></label> everywhere?

Collect labels into the CouchDB document

  • Example document here
  • Somewhere after EnabledSlots.py#L156 we can collect the union of all labels of all MRs for all projects. We probably do not need to distinguish labels from different projects.
  • We can put the list of labels into the CouchDB document under config/metadata/labels.

Extend the matching logic when a build completes

  • this script runs in jenkins every 5 min
  • it runs essentially this code
  • in particular, here we check the available new builds against test_schedule2.xml

/cc @clemenci @maszyman @shunan @mstahl

Edited by Rosen Matev