Skip to content

Draft: Persist options to FileSummaryRecord (FSR)

Ellinor Eckstein requested to merge eleckste/persist-options-to-fsr into master

Work on https://gitlab.cern.ch/lhcb-dpa/project/-/issues/271 .

fyi @erodrigu @nskidmor :)

Persist the options used to create a .dst file to its FileSummaryRecord. The goal is to have data files that are self contained i.e. information necessary for further processing should be contained in the file itself (dbtags etc.). Useful for Automatic configuration of Analysis productions for Run3 see DaVinci#145 .

Tested locally with

Moore/run lbexec Hlt2Conf.Sprucing_tests:spruce_test_lines test_spruce_data.yaml And got this in the FSR:

... '8': '/Event/Spruce/HLT2/TISTOS/Hlt2Topo2Body/Particles', '9': '/Event/Spruce/HLT2/TISTOS/Hlt2Topo3Body/Particles'}, 'version': '0'}}}, 'LumiCounter.eventsByRun': {'counts': {'255620': 4818}, 'empty': False, 'type': 'LumiEventCounter'}, 'guid': '489064FE-00B4-11EF-9E3A-FA163E36415C', 'options': {'append_decoding_keys_to_output_manifest': True, 'auditors': [], 'buffer_events': 20000, 'compression': None, 'conddb_tag': None, 'conditions_version': 'master', 'control_flow_file': None, 'data_flow_file': None, 'data_type': 'Upgrade', 'dddb_tag': None, 'event_store': 'HiveWhiteBoard', 'event_timeout': None, 'evt_max': 5000, 'first_evt': 0, 'geometry_version': 'run3/trunk', 'histo_file': 'spruce_test_lines_histos.root', 'input_files': ['mdf:root://eoslhcb.cern.ch//eos/lhcb/wg/dpa/wp1/data/2022_1_FULL_255620_00150001_0005.raw', 'mdf:root://eoslhcb.cern.ch//eos/lhcb/wg/dpa/wp1/data/2022_1_FULL_255949_00150003_0000.raw'], 'input_manifest_file': None, 'input_process': 'Hlt2', 'input_raw_format': 0.5, 'input_type': 'RAW', 'memory_pool_size': 10485760, 'monitoring_file': None, 'msg_svc_format': '% F%35W%S %7W%R%T %0W%M', 'msg_svc_time_format': '%Y-%m-%d %H:%M:%S UTC', 'n_event_slots': 1, 'n_threads': 1, 'ntuple_file': None, 'output_file': 'spruce_test_lines_production.dst', 'output_level': 3, 'output_manifest_file': 'spruce_test_lines_production.tck.json', 'output_type': 'ROOT', 'persistreco_version': 0.0, 'phoenix_filename': None, 'preamble_algs': [], 'print_freq': 10000, 'process': 'Spruce', 'python_logging_level': 20, 'require_specific_decoding_keys': [], 'scheduler_legacy_mode': True, 'simulation': False, 'tck': None, 'use_iosvc': False, 'write_decoding_keys_to_git': True, 'xml_file_catalog': None, 'xml_summary_file': None}

This is the yaml with the options used for the local test:

data_type: 'Upgrade'
simulation: False
input_raw_format: 0.5
geometry_version: run3/trunk
conditions_version: master

process: Spruce
input_type : 'RAW'

input_files : [
  'mdf:root://eoslhcb.cern.ch//eos/lhcb/wg/dpa/wp1/data/2022_1_FULL_255620_00150001_0005.raw',
  'mdf:root://eoslhcb.cern.ch//eos/lhcb/wg/dpa/wp1/data/2022_1_FULL_255949_00150003_0000.raw'
]

evt_max : 5000
persistreco_version: 0.0
output_type : 'ROOT'
input_process: Hlt2

histo_file : 'spruce_test_lines_histos.root'
output_file : 'spruce_test_lines_production.dst'
output_manifest_file : 'spruce_test_lines_production.tck.json'

Merge request reports