Add --ignore-broken-inputs option to keep running HltEfficiencyChecker if encounter a dodgy file

FYI @rmatev interested to hear your opinion on this.

As reported awhile ago by @mwhitehe (thanks!), when you run the HltEfficiencyChecker wizard, if you come across a file that cannot be opened for whatever reason, the trigger & tupling will skip it and keep running, making a perfectly valid tuple for analysis. ApplicationMgr will however exit with a not-very-descriptive error code 2, and so our check_call command in wizard.py will throw a CalledProcessError, and HltEfficiencyChecker will stop.

This is not ideal behaviour though:

• You may consider the tuple you've made to be perfectly valid since the dodgy file was simply skipped,
• The error code doesn't easily tell you what the problem was - the rest of the log shows the successful completion of a job,
• Grid sites often go down - since the tupling & trigger will naturally just keep going, why can't HltEfficiencyChecker keep going.

I stop short of enforcing any behaviour, and print a more descriptive message if we encounter this. I added the --ignore-broken-inputs arg so the user can get HltEfficiencyChecker to keep going if that is what they wish.

approved this merge request

your approach looks good!

Thanks!

unmarked as a Work In Progress

assigned to @msaur and unassigned @rjhunter

@msaur this one is ready for testing too whenever it is convenient for you.

I should make the pipeline pass first...

added 1 commit

• 4759c00c - Fixed formatting

Compare with previous version

/ci-test --merge

Started reference and integration test builds. Once done, check the comparison of build and test results.

Throughput Test Moore_hlt2_reco_baseline: 142.1 Events/s -- change of -0.15% vs. reference

Throughput Test Moore_hlt1_pp_default: 29760.0 Events/s -- change of -0.20% vs. reference

Old ci-test never finished, running new one.

/ci-test --merge

added ci-test-triggered label

• [2020-11-17 12:20] Validation started with lhcb-master-mr#1587
• [2020-11-19 00:16] Validation started with lhcb-run3-cleanup#72
• [2020-11-19 00:19] Validation started with lhcb-head#2773
• [2020-11-19 00:21] Validation started with lhcb-head-2#71
• [2020-11-19 00:25] Validation started with lhcb-gaudi-head#2786

added all-slots label

mentioned in commit afba5eb7

merged

mentioned in issue Moore#224 (closed)

added enhancement label

mentioned in commit DaVinci@48b56947