Skip to content

Sprucing dev

Nicole Skidmore requested to merge SprucingDev into master

To go with LHCb!3843 (merged), Analysis!927 (merged), DaVinci!778 (merged)

  • Use options arg so that lbexec works
  • Use root_copy_input_writer for Sprucing MC jobs so that input locations are auto propagated
  • Move output checker functions to check_output.py to remove code duplication
  • Change keyword process to input_process for all data reading functionality. Leave process for running Moore jobs
  • Sprucing now nominally uses lbexec and apply_bind. There are tests using gaudirun.py where binds are done manually
  • Put in some things eg. ProcessTypes that will hopefully pave the way for a full adoption of apply_binds in Moore
  • Propagate required MC locs in Sprucing excl. and passthrough with unit tests checking this
  • Replace MDF with RAW for lbexec use. Also online - what to do with that? - separate issue now #488
  • Move unit tests over to using lbexec
  • Move to using apply_bind
  • Decide what to do with passthrough jobs regarding process and apply_binds
  • Tests for sim, not sim, excl, pass for lbexec and gaudirun
  • process vs input_process key words in LHCb and Moore
  • process vs input_process key words in DV / Analysis
  • Why cant I print options.process in moore_control_flow? - cos using gaudirun.py
  • Change docs and tutorials for above
  • Allow HLT1 analytics tests. Separate issue https://gitlab.cern.ch/lhcb-dpa/project/-/issues/217
  • Remove redundant MC locs that get passed as input_leaves.
  • Change names of qmt tests as they are confusing
  • Make an lbexec yaml folder to separate it from the gaudirun.py options files
  • Check manually results of CI
  • Check sprucing ones with a full DEBUG
  • Final check of changes

Tests still using gaudirun

  • test_spruce_all_lines_analytics.qmt
  • Dedicated *_gaudirun.qmt tests

Note the throughput and size/rate auto tests are independent of the qmt tests so as long as the .py are unmodified (as they are) its fine

  • Check this

MC in Sprucing

  • DV expects MC particles/vertices and relations under /Event/Spruce/HLT2/.... See here

'/Event/HLT2/MC/Particles'==/Event/Spruce/HLT2/MC/Particles'

Notes for lbexec:

    ./run lbexec Hlt.Hlt2Conf.options.streaming.spruce_test_streaming:main Hlt/Hlt2Conf/options/streaming/spruce_test_streaming.yaml

Note to use gaudirun.py:

  • Take contents of spruce_test_streaming.yaml and make of the form options.input_files=, save it as a .py with 'from Moore import options` at the top
  • In this file, remove the function (and the return statement) around
    with reconstruction.bind(from_file=True, spruce=True):
        config = run_moore(options, make_streams, public_tools)
  • Then run with ./run gaudirun.py spruce_test_streaming_options.py spruce_test_streaming.py
Edited by Nicole Skidmore

Merge request reports