Skip to content

[master] Support creating production requests from a YAML file

Chris Burr requested to merge cburr/LHCbDIRAC:yaml-production-submission into master

This merge request allows production requests and steps to be generated from a YAML file using a new CLI command:

dirac-production-submit example-prod.yaml --submit
Click to expand to see example YAML file for a MC production! 
- type: Simulation
  priority: 1a
  name: CB Testing 2015 MagUp
  mc_config_version: '2015'
  sim_condition: Beam6500GeV-2015-MagUp-Nu1.6-25ns-Pythia8
  author: cburr
  inform:
  - cburr
  wg: Charm
  event_types:
    - id: 27163001
      num_events: 20000
    - id: 27163003
      num_events: 20000
  steps:
  - name: Sim09j - 2015 - MagUp - Pythia8
    processing_pass: Sim09j-ReDecay
    visible: true
    application: Gauss/v49r19
    options:
    - $APPCONFIGOPTS/Gauss/Beam6500GeV-mu100-2015-nu1.6.py
    - $APPCONFIGOPTS/Gauss/EnableSpillover-25ns.py
    - $APPCONFIGOPTS/Gauss/DataType-2015.py
    - $APPCONFIGOPTS/Gauss/RICHRandomHits.py
    - $DECFILESROOT/options/@{eventType}.py
    - $APPCONFIGOPTS/Gauss/ReDecay-100times.py
    - $APPCONFIGOPTS/Gauss/ReDecay-FullGenEventCutTool-fix.py
    - $LBPYTHIA8ROOT/options/Pythia8.py
    - $APPCONFIGOPTS/Gauss/G4PL_FTFP_BERT_EmNoCuts.py
    data_pkgs:
    - AppConfig.v3r404
    - Gen/DecFiles.v30r54
    dbtags:
      DDDB: dddb-20170721-3
      CondDB: sim-20161124-vc-mu100
    input: []
    output:
    - type: SIM
      visible: false

  - name: Digi14c for 2015+spillover
    processing_pass: Digi14c
    visible: false
    application: Boole/v30r4
    options:
    - $APPCONFIGOPTS/Boole/Default.py
    - $APPCONFIGOPTS/Boole/EnableSpillover.py
    - $APPCONFIGOPTS/Boole/DataType-2015.py
    - $APPCONFIGOPTS/Boole/Boole-SetOdinRndTrigger.py
    data_pkgs:
    - AppConfig.v3r404
    input:
    - type: SIM
      visible: false
    output:
    - type: DIGI
      visible: false
  - name: L0 emulation for 2015 - TCK 0x00a2
    processing_pass: L0Trig0x00a2
    visible: false
    application: Moore/v24r2
    options:
    - $APPCONFIGOPTS/L0App/L0AppSimProduction.py
    - $APPCONFIGOPTS/L0App/L0AppTCK-0x00a2.py
    - $APPCONFIGOPTS/L0App/ForceLUTVersionV8.py
    - $APPCONFIGOPTS/L0App/DataType-2015.py
    - $APPCONFIGOPTS/Persistency/Compression-ZLIB-1.py
    data_pkgs:
    - AppConfig.v3r404
    input:
    - type: DIGI
      visible: false
    output:
    - type: DIGI
      visible: false
    options_format: l0app
  - name: TCK-0x411400a2 Flagged for 2015
    processing_pass: Trig0x411400a2
    visible: true
    application: Moore/v24r2
    options:
    - $APPCONFIGOPTS/Moore/MooreSimProductionForSeparateL0AppStep.py
    - $APPCONFIGOPTS/Conditions/TCK-0x411400a2.py
    - $APPCONFIGOPTS/Moore/DataType-2015.py
    - $APPCONFIGOPTS/Persistency/Compression-ZLIB-1.py
    data_pkgs:
    - AppConfig.v3r404
    input:
    - type: DIGI
      visible: false
    output:
    - type: DIGI
      visible: false
  - name: Reco15a for MC 2015
    processing_pass: Reco15a
    visible: true
    application: Brunel/v48r3
    options:
    - $APPCONFIGOPTS/Brunel/DataType-2015.py
    - $APPCONFIGOPTS/Brunel/MC-WithTruth.py
    - $APPCONFIGOPTS/Persistency/Compression-ZLIB-1.py
    data_pkgs:
    - AppConfig.v3r404
    - Det/SQLDDDB.v7r10
    input:
    - type: DIGI
      visible: false
    output:
    - type: DST
      visible: False
  - name: Turbo lines (MC) for 2015 Turbo02
    processing_pass: Turbo02
    visible: true
    application: DaVinci/v40r1p3
    options:
    - $APPCONFIGOPTS/Turbo/Tesla_AllHlt2Lines_v10r0_0x00fa0051.py
    - $APPCONFIGOPTS/Turbo/Tesla_Simulation_2015_PVHLT2.py
    data_pkgs:
    - AppConfig.v3r404
    - TurboStreamProd.v2r0
    input:
    - type: DST
      visible: False
    output:
    - type: DST
      visible: False
    options_format: Tesla
  - name: Stripping24r2NoPrescalingFlagged for 2015
    processing_pass: Stripping24r2NoPrescalingFlagged
    visible: true
    application: DaVinci/v44r10p5
    options:
    - $APPCONFIGOPTS/DaVinci/DV-Stripping24r2-Stripping-MC-NoPrescaling-DST.py
    - $APPCONFIGOPTS/DaVinci/DV-RedoCaloPID-Stripping_28_24.py
    - $APPCONFIGOPTS/DaVinci/DataType-2015.py
    - $APPCONFIGOPTS/DaVinci/InputType-DST.py
    - $APPCONFIGOPTS/DaVinci/DV-Stripping-MC-muDST.py
    - $APPCONFIGOPTS/DaVinci/DV-RawEventJuggler-0_3-to-4_3.py
    data_pkgs:
    - AppConfig.v3r404
    input:
    - type: DST
      visible: False
    output:
    - type: ALLSTREAMS.MDST
      visible: False
  - name: Merge for ALLSTREAMS.MDST
    processing_pass: merging
    visible: false
    application: DaVinci/v45r6
    options:
    - $APPCONFIGOPTS/Merging/DVMergeDST.py
    - $APPCONFIGOPTS/DaVinci/DataType-2018.py
    - $APPCONFIGOPTS/Merging/WriteFSR.py
    - $APPCONFIGOPTS/Merging/MergeFSR.py
    - $APPCONFIGOPTS/Persistency/Compression-LZMA-4.py
    - $APPCONFIGOPTS/DaVinci/Simulation.py
    data_pkgs:
    - AppConfig.v3r404
    input:
    - type: ALLSTREAMS.MDST
      visible: False
    output:
    - type: ALLSTREAMS.MDST
      visible: True
    options_format: merge
  fast_simulation_type: ReDecay

This is intended to replace the create/update functionality of LHCbWebDIRAC. The main motivation being that using the webapp to create productions is error prone, tedious and inflexible.

This functionality won't be used directly by users instead:

  • For simulation LbMCSubmit will be used to generate this file from a simplier file format.
    • The tools for doing this will be maintained by Simulation-WPP.
    • Experts can customise this file for really special productions if needed to recover the flexibilty of the current webapp
  • For DPA-WP2 (analysis productions) this will be be generated automatically for users and the submission will be ran by CI
  • For special production requests (like Stripping, Sprucing, Reconstion of run 1+2) this format can be written by hand

There is still functionality to add that will come in follow up pull requests, such as:

  • the ability to run tests of a production YAML locally
  • the ability to export an existing production as a YAML file

I've already ran a test MC production with this in certification. I'd rather not add tests and full documentation just yet as the format might change slightly and I'd rather write them correctly once rather than having them be wrong.

Longer term moving to this format will give us the flexibility to simplify how productions are managed internally and replace the current webapp specific interfaces with something which handles this format directly. For now the translation is handled by ModelCompatibility module.

BEGINRELEASENOTES

*ProductionManagement NEW: Add dirac-production-submit command for creating production requests from a YAML file

ENDRELEASENOTES

Relates to https://gitlab.cern.ch/lhcb-dpa/project/-/issues/31

Edited by Chris Burr

Merge request reports