Corrupted RawEvent bank(s) when reading MDF and writing DST

When an MDF is read and the RawEvent is written in a DST, it seems that the first "DAQ" raw bank is always corrupted in the output. It is unclear if other banks may be affected.

This problem was spotted in a Moore test after adding an extra assert in RawEvent::MapView (see !4285 (comment 7100647)). However the issue can be easily reproduced with LHCb only with the (read_)dst_from_mdf tests. I've extended them in !4285 (7430179b) so that they fail right now.

As setup, the tests shows a couple of extra (secondary) problems

A memory leak related to skipping events. This may or may not be related to the fact that the skipped events are compressed.
- fixed as part of the same commit
Only 82 out of 100 events have the DAQ bank in the raw event (as seen by RawEventDump in the dst_from_mdf test before writing).
- this was understood to be because there are TAE events (we don't get a DAQ bank for events with tae window != 0)

This may be related to #114 (closed)

/cc @frankm @graven @sesen @jonrob @dcampora @nskidmor

Debugging

The sanitizers do not show anything useful (even when removing all suppressions), besides the leak

Setting a watch (write) and rwatch (read) breakpoint on RawEvent::m_banks[0].m_buff (where the DAQ bank header+payload is supposed to be) hits the breakpoints in the following (bad) order:

write! (twice)

#2  0x00007f4007ebc9c3 in LHCb::RawDataAddress::~RawDataAddress (this=0x60f00010bbb0, __in_chrg=<optimized out>) at ../DAQ/MDF/include/MDF/RawDataAddress.h:59
#3  0x00007f4007ebce5b in LHCb::RawDataAddress::~RawDataAddress (this=0x60f00010bbb0, __in_chrg=<optimized out>) at ../DAQ/MDF/include/MDF/RawDataAddress.h:62
#4  0x00007f4065928aeb in GenericAddress::release (this=0x60f00010bbb0) at ../GaudiKernel/include/GaudiKernel/GenericAddress.h:73
#5  0x00007f406f182727 in DataSvcHelpers::RegistryEntry::setAddress (this=0x60d0001ee2b0, pAddress=0x60f0000e38a8) at ../GaudiKernel/src/Lib/RegistryEntry.cpp:124
#6  0x00007f403075f407 in OutputStream::writeObjects (this=0x623000140500) at ../GaudiCommonSvc/src/PersistencySvc/OutputStream.cpp:198
#7  0x00007f4030764445 in OutputStream::execute (this=0x623000140500) at ../GaudiCommonSvc/src/PersistencySvc/OutputStream.cpp:166
#8  0x00007f403043688e in Gaudi::details::LegacyAlgorithmAdapter::execute (this=this@entry=0x623000140500) at ../GaudiKernel/include/GaudiKernel/Algorithm.h:48
#9  0x00007f406dfd2ba7 in Gaudi::Algorithm::sysExecute (this=0x623000140500, ctx=...) at ../GaudiKernel/src/Lib/Algorithm.cpp:352
#10 0x00007f406e608221 in MinimalEventLoopMgr::executeEvent (this=this@entry=0x61f00009ca80, context=...) at ../GaudiKernel/src/Lib/MinimalEventLoopMgr.cpp:350
#11 0x00007f4065810a64 in EventLoopMgr::executeEvent (this=0x61f00009ca80, ctx=...) at ../GaudiCoreSvc/src/ApplicationMgr/EventLoopMgr.cpp:259
#12 0x00007f4065814955 in EventLoopMgr::nextEvent (this=0x61f00009ca80, maxevt=<optimized out>) at ../GaudiCoreSvc/src/ApplicationMgr/EventLoopMgr.cpp:341
#13 0x00007f406e5fe80e in MinimalEventLoopMgr::executeRun (this=0x61f00009ca80, maxevt=1) at ../GaudiKernel/src/Lib/MinimalEventLoopMgr.cpp:306
#14 0x00007f40656fa2e0 in ApplicationMgr::executeRun (this=0x62400036c100, evtmax=<optimized out>) at ../GaudiCoreSvc/src/ApplicationMgr/ApplicationMgr.cpp:844
#15 0x00007f406e0cedd5 in Gaudi::Application::run (this=0x602000004b90) at ../GaudiKernel/src/Lib/Application.cpp:86

read!

#0  0x00007f4061a5803a in TBufferFile::WriteFastArray (this=0x60e00024fae0, ii=0x62e000030400, n=14)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TBufferFile.cxx:2032
#1  0x00007f4061a5eab9 in TBufferFile::WriteFastArray (this=0x60e00024fae0, i=0x62e000030400, n=14)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/inc/TBufferFile.h:524
#2  0x00007f4061cdff8b in TStreamerInfo::WriteBufferAux<TVirtualCollectionProxy> (this=0x6110002f1ec0, b=..., arr=..., compinfo=0x60c000010498, first=0, last=1, narr=621, eoffset=0,
    arrayMode=1) at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TStreamerInfoWriteBuffer.cxx:340
#3  0x00007f4061b755d1 in TStreamerInfoActions::GenericLooper::GenericWrite (buf=..., loopconf=0x6040002fa590, config=0x60c000010480)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TStreamerInfoActions.cxx:2574
#4  0x00007f4061a5ed31 in TStreamerInfoActions::TConfiguredAction::operator() (this=0x6060009a6720, buffer=..., start_collection=0x7ffceedf0ec0, end_collection=0x7ffceedf0eb0,
    loopconf=0x6040002fa590) at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/inc/TStreamerInfoActions.h:131
#5  0x00007f4061a5cc8b in TBufferFile::ApplySequence (this=0x60e00024fae0, sequence=..., start_collection=0x7ffceedf0ec0, end_collection=0x7ffceedf0eb0)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TBufferFile.cxx:3646
#6  0x00007f4061cda83e in TStreamerInfo::WriteBufferAux<char**> (this=0x6110002f1c40, b=..., arr=@0x7ffceedf1910: 0x7ffceedf1908, compinfo=0x6040002fa268, first=0, last=1, narr=1,
    eoffset=0, arrayMode=0) at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TStreamerInfoWriteBuffer.cxx:597
#7  0x00007f4061b50f20 in TStreamerInfoActions::GenericWriteAction (buf=..., addr=0x61200056c140, config=0x6040002fa250)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TStreamerInfoActions.cxx:202
#8  0x00007f4061a5ecad in TStreamerInfoActions::TConfiguredAction::operator() (this=0x6060009a6360, buffer=..., object=0x61200056c140)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/inc/TStreamerInfoActions.h:123
#9  0x00007f4061a5c994 in TBufferFile::ApplySequence (this=0x60e00024fae0, sequence=..., obj=0x61200056c140)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/io/io/src/TBufferFile.cxx:3580
#10 0x00007f40391cdeb3 in TBranchElement::FillLeavesMember (this=0x619007713780, b=...)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/tree/tree/src/TBranchElement.cxx:1763
#11 0x00007f40391bc62b in TBranch::FillImpl (this=0x619007713780, imtHelper=0x0)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/tree/tree/src/TBranch.cxx:893
#12 0x00007f40391cc6a5 in TBranchElement::FillImpl (this=0x619007713780, imtHelper=0x0)
    at /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.28.00/src/ROOT/6.28.00/tree/tree/src/TBranchElement.cxx:1265
#13 0x00007f4039df70e9 in TBranch::Fill (this=0x619007713780) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.28.00-a8528/x86_64-centos7-gcc12-dbg/include/TBranch.h:201
#14 Gaudi::RootDataConnection::save (this=this@entry=0x615000a4e780, section=..., cnt=..., cl=cl@entry=0x616000867f80, pObj=0x61200056c140, minBufferSize=2048, maxBufferSize=104857600,
    approxEventsPerBasket=<optimized out>, split_lvl=0, fill_missing=<optimized out>) at ../RootCnv/src/RootDataConnection.cpp:517
#15 0x00007f4039df8ca4 in Gaudi::RootDataConnection::saveObj (this=this@entry=0x615000a4e780, section=..., cnt=..., cl=cl@entry=0x616000867f80, pObj=<optimized out>,
    minBufferSize=<optimized out>, maxBufferSize=<optimized out>, approxEventsPerBasket=<optimized out>, split_lvl=<optimized out>, fill=<optimized out>)
    at ../RootCnv/src/RootDataConnection.cpp:484
#16 0x00007f4039d2179d in Gaudi::RootCnvSvc::i__createRep (this=0x620000017080, pObj=0x61200056c140, refpAddr=<optimized out>) at ../RootCnv/src/RootCnvSvc.cpp:392
#17 0x00007f4039d3dd80 in Gaudi::RootConverter::createRep (this=<optimized out>, pObj=0x61200056c140, refpAddr=@0x7ffceedf2520: 0x0) at ../RootCnv/include/RootCnv/RootConverter.h:92
#18 0x00007f406e1c50e6 in ConversionSvc::makeCall (this=this@entry=0x620000017080, typ=typ@entry=4, ignore_add=ignore_add@entry=true, ignore_obj=ignore_obj@entry=false,
    update=update@entry=false, pAddress=@0x7ffceedf2520: 0x0, pObject=<optimized out>) at ../GaudiKernel/src/Lib/ConversionSvc.cpp:70
#19 0x00007f406e1c6b7c in ConversionSvc::createRep (this=0x620000017080, pObj=0x61200056c140, refpAddress=<optimized out>) at ../GaudiKernel/src/Lib/ConversionSvc.cpp:121
#20 0x00007f403075f1f3 in OutputStream::writeObjects (this=0x623000140500) at ../GaudiCommonSvc/src/PersistencySvc/OutputStream.cpp:192
#21 0x00007f4030764445 in OutputStream::execute (this=0x623000140500) at ../GaudiCommonSvc/src/PersistencySvc/OutputStream.cpp:166
#22 0x00007f403043688e in Gaudi::details::LegacyAlgorithmAdapter::execute (this=this@entry=0x623000140500) at ../GaudiKernel/include/GaudiKernel/Algorithm.h:48
#23 0x00007f406dfd2ba7 in Gaudi::Algorithm::sysExecute (this=0x623000140500, ctx=...) at ../GaudiKernel/src/Lib/Algorithm.cpp:352
#24 0x00007f406e608221 in MinimalEventLoopMgr::executeEvent (this=this@entry=0x61f00009ca80, context=...) at ../GaudiKernel/src/Lib/MinimalEventLoopMgr.cpp:350
#25 0x00007f4065810a64 in EventLoopMgr::executeEvent (this=0x61f00009ca80, ctx=...) at ../GaudiCoreSvc/src/ApplicationMgr/EventLoopMgr.cpp:259
#26 0x00007f4065814955 in EventLoopMgr::nextEvent (this=0x61f00009ca80, maxevt=<optimized out>) at ../GaudiCoreSvc/src/ApplicationMgr/EventLoopMgr.cpp:341
#27 0x00007f406e5fe80e in MinimalEventLoopMgr::executeRun (this=0x61f00009ca80, maxevt=1) at ../GaudiKernel/src/Lib/MinimalEventLoopMgr.cpp:306
#28 0x00007f40656fa2e0 in ApplicationMgr::executeRun (this=0x62400036c100, evtmax=<optimized out>) at ../GaudiCoreSvc/src/ApplicationMgr/ApplicationMgr.cpp:844
#29 0x00007f406e0cedd5 in Gaudi::Application::run (this=0x602000004b90) at ../GaudiKernel/src/Lib/Application.cpp:86

Edited Sep 14, 2023 by Rosen Matev

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information