Initial lbexec support

When reviewing this merge request the individual commits likely make the most sense. In summary:

• 6843d61b Adds the two needed parts for lbexec, more specifically:
  • A DaVinci.Options class which is used internally by lbexec to parse the YAML
  • A DaVinci.make_config function which is the nominal way people should be using DaVinci
• 931f5149 makes it so that the global DaVinci.options object is never relied upon (as it will be removed once the transition to lbexec is complete)
• 7d2f007b contains an example of porting to lbexec with the test_davinci_tupling_All test. If everyone is happy with it I'll port the remaining tests and start cleaning up.

To see how this might actually look in reality, make a yaml file named options.yaml containing (most of these keys probably aren't needed):

input_files:
    - root://eoslhcb.cern.ch//eos/lhcb/wg/dpa/wp3/tests/spruce_all_lines_realtimereco_newPacking.dst'
annsvc_config: 'root://eoslhcb.cern.ch//eos/lhcb/wg/dpa/wp3/tests/spruce_all_lines_realtime_newPacking.tck.json'
data_type: Upgrade
input_type: ROOT
simulation: true
conddb_tag: sim-20171127-vc-md100
dddb_tag: dddb-20171126
histo_file: 'sprucing_histos.root'
input_raw_format: 0.3
lumi: false
ntuple_file: 'sprucing_tuple.root'
print_freq: 1
process: 'Spruce'
stream: 'default'

You can then run davinci using: lbexec DaVinciExamples.tupling.AllFunctors:alg_config options.yaml

I think it would be good to sit down and discuss what should be stored in the options.yaml and what the arguments to DaVinci.make_config should be. In particular I think things like process/stream/lumi/evt_pre_filters might be worth moving to the Python side.

Work towards DPA grand task https://gitlab.cern.ch/lhcb-dpa/project/-/issues/180 on configuration. Necessary for lhcb-dpa/analysis-productions/LbAnalysisProductions#68 (closed).

changed the description

added Configuration label

requested review from @dfazzini and @pkoppenb

changed the description

It is looking nice!

Thumbs up that you're introducing type checking in new pieces! That's something that's been on my wish-list for the longer term, and no harm in starting "today", obviously.

I hope this can be reviewed, tested and merged asap so that we "finalise" on lbexec in time for the LHCb week reports.

approved this merge request

approved this merge request

/ci-test

Started integration test build. Once done, check the results or the comparison to a reference build.

This will need to be tested again as it had non-trivial conflicts with the master branch.

/ci-test

Started integration test build. Once done, check the results or the comparison to a reference build.

Those tests are going to fail and you were faster than my local build was at failing

It should be okay now so:

/ci-test

Started integration test build. Once done, check the results or the comparison to a reference build.

added ci-test-triggered label

• [2022-06-03 13:45] Validation started with lhcb-master-mr#4626
• [2022-06-03 16:56] Validation started with lhcb-master-mr#4633
• [2022-06-03 17:15] Validation started with lhcb-master-mr#4634

added 8 commits

• 7d2f007b...a4653a36 - 4 commits from branch master
• cbf2a6c4 - Add support for lbexec
• 5bd2da88 - Fix mocking of stdin for click
• 156722f8 - Remove reliance on global state for DaVinci.options object
• 025c0085 - Move test_davinci_tupling_All to use lbexec

Compare with previous version

resolved all threads

added 2 commits

• 1893cb0f - Remove reliance on global state for DaVinci.options object
• 82c050e5 - Move test_davinci_tupling_All to use lbexec

Compare with previous version

mentioned in commit 524eb7e1

merged