Skip to content

Missing/Empty HltPackedData rawbank seems to cause algorithm failures downstream during DaVinci/v63r4 funtupling over VeloOpen TURBO

Continuing discussion from https://mattermost.web.cern.ch/lhcb/pl/8rqdu3hzriy8tcot4ymwqhb38c

Example job and log file of failure: https://lhcb-analysis-productions.web.cern.ch/pipelines/?id=5085700&prod=EMTF_CharmProductionXSec&job=DATA22_VeloOpen_DspToKmKpPip_Tuple

options: lhcb-datapkg/AnalysisProductions!389 (diffs)

Reproducer:

git clone --branch roneil-veloopen-data22 https://:@gitlab.cern.ch:8443/lhcb-datapkg/AnalysisProductions.git
cd AnalysisProductions
curl -L -O https://gitlab.cern.ch/lhcb/Moore/uploads/041efec0ec3b96f85b335f11cb6ce514/debug_veloopen.yaml

# the failure happens after 70k events
lb-run -c "x86_64_v3-centos7-gcc11+dd4hep-opt+g" DaVinci/v63r4 lbexec EMTF_CharmProductionXSec.DaVinci.mesons:DspToKmKpPip_alg_config debug_veloopen.yaml

debug_veloopen.yaml

snippet of job output

HltPackedBufferDecoder#1            WARNING No HltPackedData raw bank (the DstData bank) in raw event.
ParticleUnpacker                      ERROR Input has empty map of buffers -- can not do anything at this point. Verify HltPackedBufferDecoder configuration
RecVertexUnpacker                     ERROR Input has empty map of buffers -- can not do anything at this point. Verify HltPackedBufferDecoder configuration
RecV1ToPVConverter                    ERROR RecV1ToPVConverter : Cannot retrieve /Event/HLT2/Rec/Vertex/Primary from transient store.
RecV1ToPVConverter                    ERROR Maximum number of errors ( 'ErrorMax':1) reached.
HLTControlFlowMgr                     FATAL Event failed in Node FunTupleBase_Particles/DspToKmKpPip : Error in algorithm execute
HLTControlFlowMgr                     FATAL *** Event 7893 on slot 0 failed! ***
HLTControlFlowMgr                      INFO Dumping Alg Exec State for slot 0:
  [slot: 0, incident: AlgFail]:

  +        Hlt1DecReportsDecoder#1  e: d f: 1 sc: Filter decision: PASSED
  +       HltPackedBufferDecoder#1  e: d f: 1 sc: Filter decision: PASSED
  +         LHCb__UnpackRawEvent#5  e: d f: 1 sc: Filter decision: PASSED
  +               ParticleUnpacker  e: d f: 1 sc: Filter decision: PASSED
  +             RecV1ToPVConverter  e: d f: 1 sc: FAILURE
  +                     createODIN  e: d f: 1 sc: Filter decision: PASSED
  + Gaudi__Hive__FetchDataFromFile  e: d f: 1 sc: SUCCESS
  +         LHCb__UnpackRawEvent#4  e: d f: 1 sc: Filter decision: PASSED
  +         HDRFilter_DspToKmKpPip  e: d f: 0 sc: Filter decision: PASSED
  +               ReserveIOVDD4hep  e: d f: 1 sc: Filter decision: PASSED
  +                   DspToKmKpPip  e: d f: 1 sc: Filter decision: PASSED
  +              RecVertexUnpacker  e: d f: 1 sc: Filter decision: PASSED
  +           LHCb__UnpackRawEvent  e: d f: 1 sc: Filter decision: PASSED
  +        Hlt2DecReportsDecoder#1  e: d f: 1 sc: Filter decision: PASSED

HLTControlFlowMgr                     FATAL Failed event detected on s: 0  e: 7893
HLTControlFlowMgr                     FATAL *** Event 7894 on slot 0 failed! ***
HLTControlFlowMgr                      INFO Dumping Alg Exec State for slot 0:
  [slot: 0, incident: AlgFail]:

  +        Hlt1DecReportsDecoder#1  e: d f: 1 sc: Filter decision: PASSED
  +       HltPackedBufferDecoder#1  e: d f: 1 sc: Filter decision: PASSED
  +         LHCb__UnpackRawEvent#5  e: d f: 1 sc: Filter decision: PASSED
  +               ParticleUnpacker  e: d f: 1 sc: Filter decision: PASSED
  +             RecV1ToPVConverter  e: d f: 1 sc: FAILURE
  +                     createODIN  e: d f: 1 sc: Filter decision: PASSED
  + Gaudi__Hive__FetchDataFromFile  e: d f: 1 sc: SUCCESS
  +         LHCb__UnpackRawEvent#4  e: d f: 1 sc: Filter decision: PASSED
  +         HDRFilter_DspToKmKpPip  e: d f: 0 sc: Filter decision: FAILED
  +               ReserveIOVDD4hep  e: d f: 1 sc: Filter decision: PASSED
  +                   DspToKmKpPip  e: d f: 1 sc: Filter decision: PASSED
  +              RecVertexUnpacker  e: d f: 1 sc: Filter decision: PASSED
  +           LHCb__UnpackRawEvent  e: d f: 1 sc: Filter decision: PASSED
  +        Hlt2DecReportsDecoder#1  e: d f: 1 sc: Filter decision: PASSED

HLTControlFlowMgr                     FATAL Failed event detected on s: 0  e: 7894
HLTControlFlowMgr                     FATAL *** Event 7895 on slot 0 failed! ***
HLTControlFlowMgr                      INFO Dumping Alg Exec State for slot 0:
  [slot: 0, incident: AlgFail]:

  +        Hlt1DecReportsDecoder#1  e: d f: 1 sc: Filter decision: PASSED
  +       HltPackedBufferDecoder#1  e: d f: 1 sc: Filter decision: PASSED
  +         LHCb__UnpackRawEvent#5  e: d f: 1 sc: Filter decision: PASSED
  +               ParticleUnpacker  e: d f: 1 sc: Filter decision: PASSED
  +             RecV1ToPVConverter  e: d f: 1 sc: Filter decision: PASSED
  +                     createODIN  e: d f: 1 sc: Filter decision: PASSED
  + Gaudi__Hive__FetchDataFromFile  e: d f: 1 sc: SUCCESS
  +         LHCb__UnpackRawEvent#4  e: d f: 1 sc: Filter decision: PASSED
  +         HDRFilter_DspToKmKpPip  e: d f: 0 sc: Filter decision: PASSED
  +               ReserveIOVDD4hep  e: d f: 1 sc: Filter decision: PASSED
  +                   DspToKmKpPip  e: d f: 1 sc: Filter decision: PASSED
  +              RecVertexUnpacker  e: d f: 1 sc: Filter decision: PASSED
  +           LHCb__UnpackRawEvent  e: d f: 1 sc: Filter decision: PASSED
  +        Hlt2DecReportsDecoder#1  e: d f: 1 sc: Filter decision: PASSED

cc @sstahl

Edited by Ryunosuke O'Neil