Skip to content
Snippets Groups Projects

Failed job is marked as success due to FSRs

    • Closed Issue created by Nicole Skidmore @nskidmor

    When running the Sprucing followed by a file merging step the summaryMoore_<>.xml for the merging step reports

    <success>True</success>

    but the log shows the file to merged was not found

    EventSelector                          INFO Stream:EventSelector.DataStreamTool_1 Def:DATAFILE='LFN:00012345_00006789_1.rd.dst' SVC='LHCb::MDFSelecto
r' OPT='READ' IgnoreChecksum='YES'
TFile::TFile                          ERROR file /afs/cern.ch/work/n/nskidmor/public/Sprucing_production/Local_qc_24359_JobDir/LFN:00012345_00006789_
1.rd.dst does not exist

    Minimal reproducers

    merge.py 
    from GaudiConf.LbExec import Options
from PyConf.application import configure_input, configure, root_copy_input_writer
from PyConf.Algorithms import Gaudi__Hive__FetchLeavesFromFile
from PyConf.control_flow import CompositeNode, NodeLogic


def dst(options: Options):
    input_leaves = Gaudi__Hive__FetchLeavesFromFile()
    writer = root_copy_input_writer(options.output_file, input_leaves)
    node = CompositeNode(
        'DSTMerger',
        combine_logic=NodeLogic.LAZY_AND,
        children=[input_leaves, writer],
        force_order=True)

    config = configure_input(options)
    config.update(configure(options, node))
    return config
    prod.yaml 
    - author: nskidmor
  name: Passthrough sprucing (2022 MagDown)
  comment: ''
  type: Sprucing

  wg: DPA
  inform:
  - nskidmor
  priority: 1a

  input_dataset:
    # make_conditions.py '/LHCb/Collision22/Beam6800GeV-VeloClosed-MagDown-Excl-UT/Real Data/94000000/RAW' --dq UNCHECKED
    conditions_dict:
      configName: LHCb
      configVersion: Collision22
      inFileType: RAW
      inProPass: Real Data
      inDataQualityFlag: UNCHECKED
      inProductionID: ALL
      inTCKs: ALL
    conditions_id: 434924
    conditions_description: Beam6800GeV-VeloClosed-MagDown-Excl-UT
    event_type: '94000000'

  steps:
  - name: Passthrough sprucing
    processing_pass: Pass01
    visible: true
    application:
      name: Moore
      version: v54r7
    data_pkgs: []
    options:
      entrypoint: Hlt2Conf.Sprucing_production:pass_spruce_production
      extra_options:
        input_raw_format : 0.5
        input_type : 'RAW'
        simulation : False
        data_type : 'Upgrade'
        geometry_version: trunk
        conditions_version: master
        compression: "ZSTD:1"
        output_type : 'ROOT'
        input_process: "Hlt2"
    input:
    - type: MDF
      visible: true
    output:
    - type: SL.DST
      visible: false
    - type: CHARM.DST
      visible: false
    - type: B2CC.DST
      visible: false
    - type: RD.DST
      visible: false
    - type: BANDQ.DST
      visible: false
    - type: QEE.DST
      visible: false
    - type: B2OC.DST
      visible: false
    - type: BNOC.DST
      visible: false
  - name: Pass merge
    processing_pass: PassMerge
    visible: false
    application:
      name: Moore
      version: v54r7
    data_pkgs: []
    options:
      entrypoint: /afs/cern.ch/work/n/nskidmor/public/Sprucing_production/merge.py:dst
      extra_options:
        input_raw_format : 0.5
        input_type : 'RAW'
        simulation : False
        data_type : 'Upgrade'
        geometry_version: trunk
        conditions_version: master
        compression: "ZSTD:1"
        output_type : 'ROOT'
        input_process: "Hlt2"
    input:
    - type: RD.DST
      visible: false
    output:
    - type: RD.DST
      visible: true

    Run with

    lb-dirac bash
lhcb-proxy-init
dirac-production-request-run-local prod.yaml 'Passthrough sprucing (2022 MagDown)' --input-files='/lhcb/data/2022/RAW/FULL/LHCb/COLLISION22/256170/256170_00100013_0000.raw'
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Nicole Skidmore assigned to @clemenci and @cburr

      assigned to @clemenci and @cburr

    • Nicole Skidmore changed title from Successful job fails to find input to Successful file merging job fails to find input

      changed title from Successful job fails to find input to Successful file merging job fails to find input

    • Eduardo Rodrigues assigned to @chaen

      assigned to @chaen

    • Eduardo Rodrigues added DPA-WP1 label

      added DPA-WP1 label

    • Chris Burr unassigned @chaen

      unassigned @chaen

    • Chris Burr changed title from Successful file merging job fails to find input to Failed job is marked as success due to FSRs

      changed title from Successful file merging job fails to find input to Failed job is marked as success due to FSRs

    • Chris Burr
      Chris Burr @cburr ·
      Owner

      @clemenci A simpler reproducer is to create these three files:

      merge.py 
      from GaudiConf.LbExec import Options
from PyConf.application import configure_input, configure, root_copy_input_writer
from PyConf.Algorithms import Gaudi__Hive__FetchLeavesFromFile
from PyConf.control_flow import CompositeNode, NodeLogic


def dst(options: Options):
    input_leaves = Gaudi__Hive__FetchLeavesFromFile()
    writer = root_copy_input_writer(options.output_file, input_leaves)
    node = CompositeNode(
        'DSTMerger',
        combine_logic=NodeLogic.LAZY_AND,
        children=[input_leaves, writer],
        force_order=True)

    config = configure_input(options)
    config.update(configure(options, node))
    return config
      options.yaml 
      compression: ZSTD:1
conditions_version: master
data_type: Upgrade
geometry_version: trunk
input_files:
- LFN:00012345_00006789_1.charm.dst
input_process: Hlt2
input_raw_format: 0.5
input_type: RAW
n_threads: 1
output_file: 00012345_00006789_2.charm.dst
output_type: ROOT
simulation: false
xml_file_catalog: pool_xml_catalog.xml
xml_summary_file: summaryMoore_00012345_00006789_2.xml
      pool_xml_catalog.xml 
      <?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<!DOCTYPE POOLFILECATALOG SYSTEM "InMemory">
<POOLFILECATALOG>
  <File ID="D4A6D538-F96A-11ED-862A-3CECEF1070AC">
    <physical>
      <pfn filetype="ROOT" name="00012345_00006789_1.charm.dst"/>
    </physical>
    <logical/>
  </File>
</POOLFILECATALOG>

      And then run:

      # A real DST behaves the same way but this is simplier for the reproducer
touch 00012345_00006789_1.charm.dst
lb-run --siteroot=/cvmfs/lhcb.cern.ch/lib/ -c best Moore/v54r7 lbexec merge.py:dst options.yaml

      Despite the error, this is indistingishable from a sucessful job as:

      • return code of the application is 0
      • the summary XML contains <success>True</success>
      • the output file is written
    • Chris Burr added bug label

      added bug label

    • Chris Burr unassigned @cburr

      unassigned @cburr

      • Chris Burr
        Chris Burr @cburr ·
        Owner

        Changing input_process: Hlt2 to input_process: Spruce causes the job to pass. The difference in an options dump between the two is:

        diff --git a/with-hlt2.json b/with-spruce.json
index f6a4fff..d82b6c6 100644
--- a/with-hlt2.json
+++ b/with-spruce.json
@@ -1,7 +1,7 @@
 {
     "ApplicationMgr.OutputLevel": "3",
     "ApplicationMgr.EvtMax": "-1",
-    "ApplicationMgr.ExtSvc": "['HiveWhiteBoard/EventDataSvc', 'Gaudi::MultiFileCatalog/FileCatalog', 'Gaudi::IODataManager/IODataManager', 'Gaudi::RootCnvSvc/RootCnvSvc', 'Gaudi::RootCnvSvc/FileRecordCnvSvc', 'GitANNSvc/HltANNSvc', 'Gaudi::Monitoring::MessageSvcSink/Gaudi::Monitoring::MessageSvcSink', 'LHCb::FSR::Sink/FileSummaryRecord', 'XMLSummarySvc/XMLSummarySvc']",
+    "ApplicationMgr.ExtSvc": "['HiveWhiteBoard/EventDataSvc', 'Gaudi::MultiFileCatalog/FileCatalog', 'Gaudi::IODataManager/IODataManager', 'Gaudi::RootCnvSvc/RootCnvSvc', 'Gaudi::RootCnvSvc/FileRecordCnvSvc', 'GitANNSvc/HltANNSvc', 'Gaudi::Monitoring::MessageSvcSink/Gaudi::Monitoring::MessageSvcSink', 'XMLSummarySvc/XMLSummarySvc']",
     "ApplicationMgr.EventLoop": "\"HLTControlFlowMgr/HLTControlFlowMgr\"",
     "ApplicationMgr.EvtSel": "\"EventSelector\"",
     "DDDBConf.DataType": "\"Upgrade\"",
@@ -58,7 +58,6 @@
     "HltANNSvc.Repositories": "[('/home/cburr/sprucing-validation/v2/Local_yd49mv91_JobDir/lhcb-metainfo/.git', 'master'), ('/cvmfs/lhcb-condb.cern.ch/git-conddb/file-content-metadata.git', 'master')]",
     "MessageSvc.Format": "\"% F%35W%S %7W%R%T %0W%M\"",
     "MessageSvc.timeFormat": "\"%Y-%m-%d %H:%M:%S UTC\"",
-    "FileSummaryRecord.AcceptRegex": "\"^LumiCounter\\.eventsByRun$\"",
     "XMLSummarySvc.xmlfile": "\"summaryMoore_00012345_00006789_2.xml\"",
     "XMLSummarySvc.EndEventIncident": "\"FinishedProcessingSomeEvent\"",
     "RFileCnv.GlobalCompression": "\"ZSTD:1\"",
      • Nicole Skidmore
        Nicole Skidmore @nskidmor ·
        Author Maintainer

        Thanks Chris, it seems with-hlt2.json is actually what we want ie. with the FSR sink, but that this does not work.

      • Eduardo Rodrigues
        Eduardo Rodrigues @erodrigu ·
        Developer

        For the record for those reading these endeavours - we agreed to try and get this sorted on the timescale of 2 weeks given other urgent matters & co. Everyone happy :-).

      • Please register or sign in to reply
    • C
      Marco Clemencic @clemenci ·
      Owner

      This issue should be fixed by the combination of !4138 (merged) and gaudi/Gaudi!1456 (merged).

    • Eduardo Rodrigues changed milestone to %RTA/2023.06.1

      changed milestone to %RTA/2023.06.1

    • Eduardo Rodrigues
      Eduardo Rodrigues @erodrigu ·
      Developer

      Closed via !4138 (merged) and gaudi/Gaudi!1456 (merged).

    • Eduardo Rodrigues closed

      closed

    Please register or sign in to reply