Skip to content

Unifying how Run 3 applications are run

This is a proposal for how to unify the running of Run 3 applications across both interactive and grid style running. Current usages of gaudirun.py/davinci run-mc would be replaced by a new command provided by the LHCb package, lbexec.

Motivation

This is driven by the needs of grid jobs as well as avoiding fragmentation between how applications are ran.

There are a few constraints which have lead me to this idea:

  • Provide a reliable interface for setting the configuration needed running large numbers of jobs (definitely input_files/output_filename/evt_max and preferably root_compression_level/dddb_tag/data_type/simulation/...)

  • The common part in LHCb should be minimal and leave flexibility for each application while still ensuring some configuration can be set from outside

  • One and only one solution for all applications in LHCb (Moore/Allen/DaVinci/Gauss, interactively/Ganga/DIRAC/LHCbPR)

Specification of lbexec

The arguments are:

  • First: Qualifier of the module to import and function to call
  • Second: YAML files which are used to populate $GAUDIAPPNAME.Options. Multiple YAML files can be provided here which will be overlaid before loading and LHCbDIRAC will provide a command for generating the basic options from the bookkeeping.
  • Remaining (optional): Arguments which are passed to the function as strings

The user provided function (argument 1) would always take an options argument that replaces the current global (Moore.options/DaVinci.options). The Options class would be inherited from LHCb with Moore/DaVinci extending it as needed. All properties on the options object would be loaded from the YAML and the object would then be immutable. This function would return an instance of PyConf.application.ComponentConfig, which would typically be generated a function like Moore.run_moore. I propose renaming this function to something like generate_config/make_config to avoid confusion about who is actually starting the event loop.

Examples

lbexec AppConfig.FEST.Sprucing:passthrough options.yaml
lbexec AppConfig.FEST.Sprucing:exclusive options.yaml
lbexec my_options:main options.yaml+data.yaml "DpToKee_SS"

And the corresponding function signatures (the type annotations aren't required):

from PyConf.application import ComponentConfig

from Moore import Options, run_moore
from RecoConf.reconstruction_objects import reconstruction

def passthrough(options: Options) -> ComponentConfig:
    with reconstruction.bind(from_file=True, spruce=True):
        return run_moore(options, ...)

def exclusive(options: Options) -> ComponentConfig:
    with reconstruction.bind(from_file=True, spruce=True):
        return run_moore(options, ...)


from DaVinci import Options, generate_config

def main(options: Options, decay: str) -> ComponentConfig:
    return generate_config(configured_FunTuple(...))

I have a (very) basic prototype working locally for Moore and DaVinci and I've pushed the main parts here:

Edited by Laurent Dufour
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information