Skip to content

Loading RawEvent from DST produced from MDF causes segfault

I have an application which loads an MDF and writes out the raw event to a DST:

from Configurables import ApplicationMgr, CondDB, LHCbApp, LHCb__MDFWriter, OutputStream
from GaudiConf import IOHelper
from PRConfig.TestFileDB import test_file_db

# Change this to "MDF" to write output.mdf instead of output.dst
output_type = "ROOT"
if output_type == "ROOT":
    output_file = "output.dst"
    writer = OutputStream(
        ItemList=["/Event/DAQ/RawEvent#1"],
        Output="DATAFILE='{}' SVC='Gaudi::RootCnvSvc' OPT='RECREATE'".format(output_file),
    )
else:
    output_file = "output.mdf"
    writer = LHCb__MDFWriter(
        BankLocation="/Event/DAQ/RawEvent",
        Connection="file://{}".format(output_file),
    )


LHCbApp(DataType="2018", EvtMax=1000)
f = test_file_db['MiniBrunel_2018_MinBias_FTv4_MDF']
IOHelper("MDF", output_type).inputFiles(list(set(f.filenames)))
ApplicationMgr(TopAlg=[writer])

# Don't bother checking the run number; as the MDF contains Upgrade data, older versions of
# LHCb use default tags that won't be valid for the Upgrade run numbers (using the 2018
# DataType for Upgrade data does not change the segfault outcome) 
CondDB().IgnoreHeartBeat = True
CondDB().EnableRunStampCheck = False

I run this with lb-run LHCb/v51r3 gaudirun.py app.py, and then analyse the output with a GaudiPython script, which loads the LHCb::RawEvent object and queries its banks:

from Configurables import LHCbApp
import GaudiPython as GP
from GaudiConf import IOHelper

# Change this to "MDF" to read output.mdf instead of output.dst
input_type = "ROOT"
if input_type = "ROOT":
    IOHelper('ROOT').inputFiles(["output.dst"], clear=True)
else:
    IOHelper('MDF').inputFiles(["output.mdf"], clear=True)

LHCbApp(DataType="Upgrade")

appMgr = GP.AppMgr()
TES = appMgr.evtsvc()
appMgr.run(1)

for i in range(1000):
    print(i)
    # Query the raw event
    TES["/Event/DAQ/RawEvent"].banks(GP.gbl.LHCb.RawBank.DstData)
    appMgr.run(1)

which I run with lb-run LHCb/v51r3 python explore.py.

Running the GaudiPython script triggers a segmentation fault. When running with platform x86_64-centos7-gcc9-dbg, I get:

EventSelector     SUCCESS Reading Event record 1. Record number within stream 1: 1

 *** Break *** segmentation violation



===========================================================
There was a crash (kSigSegmentationViolation).
This is the entire stack trace of all threads:
===========================================================
#0  0x00007fdea8f6f46c in waitpid () from /lib64/libc.so.6
#1  0x00007fdea8eecf62 in do_system () from /lib64/libc.so.6
#2  0x00007fde9f1f262a in TUnixSystem::Exec (this=0x1df1960, shellcmd=0x18c50320 "/cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc9-dbg/etc/gdb-backtrace.sh 20872 1>&2") at /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/core/unix/src/TUnixSystem.cxx:2107
#3  0x00007fde9f1f2e6c in TUnixSystem::StackTrace (this=0x1df1960) at /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/core/unix/src/TUnixSystem.cxx:2397
#4  0x00007fde9f1f6738 in TUnixSystem::DispatchSignals (this=0x1df1960, sig=kSigSegmentationViolation) at /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/core/unix/src/TUnixSystem.cxx:3628
#5  0x00007fde9f1ee8be in SigHandler (sig=kSigSegmentationViolation) at /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/core/unix/src/TUnixSystem.cxx:403
#6  0x00007fde9f1f668c in sighandler (sig=11) at /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/core/unix/src/TUnixSystem.cxx:3605
#7  <signal handler called>
#8  boost::container::vector<LHCb::RawBank const*, boost::container::dtl::static_storage_allocator<LHCb::RawBank const*, 350ul, 0ul, true>, void>::room_enough (this=0x18cb3c18) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/Boost/1.72.0-8f9a5/x86_64-centos7-gcc9-dbg/include/boost/container/vector.hpp:2643
#9  boost::container::vector<LHCb::RawBank const*, boost::container::dtl::static_storage_allocator<LHCb::RawBank const*, 350ul, 0ul, true>, void>::priv_push_back<LHCb::RawBank const*> (u=<optimized out>, this=0x18cb3c18) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/Boost/1.72.0-8f9a5/x86_64-centos7-gcc9-dbg/include/boost/container/vector.hpp:2645
#10 boost::container::vector<LHCb::RawBank const*, boost::container::dtl::static_storage_allocator<LHCb::RawBank const*, 350ul, 0ul, true>, void>::push_back (x=<optimized out>, this=0x18cb3c18) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/Boost/1.72.0-8f9a5/x86_64-centos7-gcc9-dbg/include/boost/container/vector.hpp:1901
#11 LHCb::RawEvent::mapBanks (this=this
entry=0x187eb920, bankType=bankType
entry=LHCb::RawBank::ODIN) at /workspace/build/LHCB/LHCB_v51r3/Event/DAQEvent/src/RawEvent.cpp:43
#12 0x00007fde5dea2109 in LHCb::RawEvent::banks (bankType=LHCb::RawBank::ODIN, this=0x187eb920) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_97a/cppgsl/3.0.1/x86_64-centos7-gcc9-dbg/include/gsl/gsl_assert:116
#13 ODINDecodeTool::execute (this=0x176dcd70) at /workspace/build/LHCB/LHCB_v51r3/Kernel/LHCbAlgs/src/ODINDecodeTool.cpp:108
#14 0x00007fde5deb5f58 in OdinTimeDecoder::getODIN (this=this
entry=0x176deef0) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/include/c++/9.2.0/bits/atomic_base.h:734
#15 0x00007fde5deb6273 in OdinTimeDecoder::getTime (this=0x176deef0) at /workspace/build/LHCB/LHCB_v51r3/Kernel/LHCbAlgs/src/OdinTimeDecoder.cpp:110
#16 0x00007fde5e392630 in EventClockSvc::handle (this=0x1769fc10, inc=...) at /cvmfs/lhcb.cern.ch/lib/lhcb/GAUDI/GAUDI_v34r1/InstallArea/x86_64-centos7-gcc9-dbg/include/GaudiKernel/SmartIF.h:82
#17 0x00007fde7fb58a9d in IncidentSvc::i_fireIncident (this=this
entry=0x1751c9f0, incident=..., listenerType=...) at /workspace/build/GAUDI/GAUDI_v34r1/GaudiCoreSvc/src/IncidentSvc/IncidentSvc.cpp:190
#18 0x00007fde7fb58ca5 in IncidentSvc::fireIncident (this=0x1751c9f0, incident=...) at /workspace/build/GAUDI/GAUDI_v34r1/GaudiKernel/GaudiKernel/Incident.h:48
#19 0x00007fde7fac97aa in EventLoopMgr::executeEvent (this=0x174ebb10, ctx=...) at /workspace/build/GAUDI/GAUDI_v34r1/GaudiKernel/GaudiKernel/SmartIF.h:82
#20 0x00007fde7facc2bc in EventLoopMgr::nextEvent (this=0x174ebb10, maxevt=1) at /workspace/build/GAUDI/GAUDI_v34r1/GaudiCoreSvc/src/ApplicationMgr/EventLoopMgr.cpp:341
#21 0x00007fde88b0e3dd in MinimalEventLoopMgr::executeRun (this=<optimized out>, maxevt=<optimized out>) at /workspace/build/GAUDI/GAUDI_v34r1/GaudiKernel/src/Lib/MinimalEventLoopMgr.cpp:306
#22 0x00007fde7fa983be in ApplicationMgr::executeRun (this=0x1581a790, evtmax=1) at /workspace/build/GAUDI/GAUDI_v34r1/GaudiCoreSvc/src/ApplicationMgr/ApplicationMgr.cpp:833
#23 0x00007fde84817047 in ?? ()
#24 0x0000000000000000 in ?? ()
===========================================================

This points to a problem in the usage of boost::container::static_vector in LHCb::RawEvent. This was added in 335073df (!2027 (merged)).

If I re-run the two steps using the latest LHCb release from run2-patches, v45r5, I cannot reproduce the failure. This is consistent with the problem being somewhere in LHCb::RawEvent, as the static_vector changes are not present in run2-patches. I cannot reproduce the failure if I write an MDF. Perhaps the issue is somewhere in the ROOT (de)serialisation of boost::container::static_vector (a pure guess)?

As far as I can tell, this is the same issue that's blocking Moore!685 (merged).

/cc @sponce @graven @rmatev

Edited by Alex Pearce
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information