Improve feedback
This is a low priority issue meant to collect and refine some ideas on how to improve the feedback on MRs.
List which MRs changed, ideally linking to a comparison page
the current message is:
Some MRs have changed since the launch of the ci-test slot. We will continue with the ref update. Correctness not guaranteed!Give a bit more detail on errors
example to improve
Error during git operations, no reference update MR created for LHCb!Don't put all feedback in code blocks
Once there's a bit more structure to the feedback, the monospace font code blocks are distracting.
Combine (some) warnings per test
currently:
Moore/x86_64_v2-centos7-gcc11-dbg: Test Hlt1Conf.test_hlt1_smog_example contains nonzero exit code and/or warnings!
Moore/x86_64_v2-centos7-gcc11-dbg: Test RecoConf.hlt2_reco_baseline_with_mcchecking contains nonzero exit code and/or warnings!
Moore/x86_64_v2-centos7-gcc11-dbg: Test without ref failed: Hlt1Conf.test_hlt1_smog_example!
Moore/x86_64_v2-centos7-gcc11-dbg: Test without ref timed out: Hlt2Conf.test_hlt2_noUT_trackefflines!
Moore/x86_64_v2-centos7-gcc11-dbg: Test without ref timed out: Hlt2Conf.test_hlt2_trackefflines!
Moore: Correctness for treatment of Hlt/RecoConf/tests/refs/hlt2_lead_lead_light_reco_pr_kf_with_mcchecking.ref not guaranteed. Test platforms with unclear statuses:
[('x86_64_v2-centos7-gcc11-dbg', 'timeout')]
Moore: Correctness for treatment of RecoConf.hlt2_light_reco_calo_efficiency not guaranteed. Test platforms with unclear statuses:
[('x86_64_v2-centos7-gcc11-dbg', 'timeout')]
Moore: Correctness for treatment of Hlt/RecoConf/tests/refs/hlt2_reco_baseline_UTTELL40_with_mcchecking.ref not guaranteed. Test platforms with unclear statuses:
[('x86_64_v2-centos7-gcc11-dbg', 'timeout')]
Moore: Correctness for treatment of Hlt/RecoConf/tests/refs/hlt2_reco_baseline_with_cheatedseeding.ref not guaranteed. Test platforms with unclear statuses:
[('x86_64_v2-centos7-gcc11-dbg', 'timeout')]
Moore: Correctness for treatment of Hlt/RecoConf/tests/refs/hlt2_reco_baseline_with_mcchecking.ref not guaranteed. Test platforms with unclear statuses:
[('x86_64_v2-centos7-gcc11-dbg', 'error')]perhaps better (should also add links and replace the ref filename with the test name):
- Test failures (without references)
- Hlt1Conf.test_hlt1_smog_example failed on x86_64_v2-centos7-gcc11-dbg, ...
- RecoConf.hlt2_reco_baseline_with_mcchecking failed on x86_64_v2-centos7-gcc11-dbg, ...
- Test timeouts
- Hlt2Conf.test_hlt2_noUT_trackefflines timed out on x86_64_v2-centos7-gcc11-dbg, ...
- Hlt2Conf.test_hlt2_trackefflines timed out on x86_64_v2-centos7-gcc11-dbg, ...
- Test timeouts (reference update may be incomplete)
- Hlt/RecoConf/tests/refs/hlt2_lead_lead_light_reco_pr_kf_with_mcchecking.ref on x86_64_v2-centos7-gcc11-dbg ...
- RecoConf.hlt2_light_reco_calo_efficiency on x86_64_v2-centos7-gcc11-dbg, ...
- Hlt/RecoConf/tests/refs/hlt2_reco_baseline_UTTELL40_with_mcchecking.ref on x86_64_v2-centos7-gcc11-dbg
- Hlt/RecoConf/tests/refs/hlt2_reco_baseline_with_cheatedseeding.ref on x86_64_v2-centos7-gcc11-dbg
- Hlt/RecoConf/tests/refs/hlt2_reco_baseline_with_mcchecking.ref on x86_64_v2-centos7-gcc11-dbgActivity
@chasse opinions welcome here.
The end goal for this effort is to get something easily comprehensible from anyone involved in the MR. Then we can split out / copy this part into its own thing and post feedback on MRs for every ci-test (independently of reference updates). My dream is of course that I never need to say "please also check this failing test" since people missed something in the nightly dashboard (which is admittedly easy to happen).
