gaudirun.py --tcmalloc causes segmentation violation in detdesc detector geometry access

cc @cburr @bcouturi @sponce

Minimal options to be ran with hlt2_pp_2025.py with instructions to create the segmentation violation in 0bc47c7b

Found when attempting to run the BW Tests on MC. The BW Tests have used this flag since 2023 and it definitely used to be able to run on MC but I have a gut feeling that these tests haven't worked on MC since we moved back from 2024-patches to master

Full segvio log

I won't attach the successful version of the log, i.e. without -T because the filesize is too large.

@cburr said:

That means there is a subtle bug somewhere, tcmalloc shouldn't make a difference but if you're relying on something which should be broken it can be a bit less permissive

added Crash label

It seems to be when loading the conditions:

#6  0x00007f7771695eed in UpdateManagerSvc::Item::addChild(BaseObjectMemberFunction*, UpdateManagerSvc::Item*) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescSvc.so
#7  0x00007f777168cd50 in UpdateManagerSvc::i_registerCondition(void*, BaseObjectMemberFunction*) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescSvc.so
#8  0x00007f776f0a9ec4 in GeometryInfoPlus::registerSupportGI() () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescLib.so
#9  0x00007f776f0abc8a in GeometryInfoPlus::initialize() () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescLib.so
#10 0x00007f776f0ad2cc in GeometryInfoPlus::GeometryInfoPlus(DetDesc::IDetectorElementPlus*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescLib.so
#11 0x00007f776f0a4d51 in GeoInfo::createGeometryInfo(DetDesc::IDetectorElementPlus*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescLib.so
#12 0x00007f776f09c34a in DetDesc::DetectorElementPlus::createGeometryInfo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescLib.so
#13 0x00007f77715723c3 in XmlBaseDetElemCnv::i_fillObj(xercesc_3_2::DOMElement*, DataObject*, IOpaqueAddress*) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescCnvLib.so
#14 0x00007f7771577fdc in XmlGenericCnv::internalCreateObj(xercesc_3_2::DOMElement*, DataObject*&, IOpaqueAddress*) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescCnvLib.so
#15 0x00007f7771579e5d in XmlGenericCnv::createObj(IOpaqueAddress*, DataObject*&) () from /home/lben/develop/master/LHCb/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libDetDescCnvLib.so
#16 0x00007f77958a8985 in ConversionSvc::makeCall(int, bool, bool, bool, IOpaqueAddress*&, DataObject*&) () from /home/lben/develop/master/Gaudi/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libGaudiKernel.so
#17 0x00007f77958a8bba in virtual thunk to ConversionSvc::createObj(IOpaqueAddress*, DataObject*&) () from /home/lben/develop/master/Gaudi/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libGaudiKernel.so
#18 0x00007f7771cca708 in PersistencySvc::makeCall(int, IOpaqueAddress*&, DataObject*&) () from /home/lben/develop/master/Gaudi/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libGaudiCommonSvc.so
#19 0x00007f7771cca8c2 in virtual thunk to PersistencySvc::createObj(IOpaqueAddress*, DataObject*&) () from /home/lben/develop/master/Gaudi/InstallArea/x86_64_v2-el9-gcc13+detdesc-opt/lib/libGaudiCommonSvc.so

I tried to debug with gdb, but too much is optimized. I am trying to build the gcc13+detdesc-do0 but this does not seem possible !

On my side, I've tried to reproduce locally in dbg mode, but I do not even manage to compile Moore these days... I have troubles with ParamFiles. It seems it was not released recently enough and you need your own local version. But even with it, I get this kind of errors :

TBTC_Forward_a7fe5a32.GhostProba...   FATAL  Standard std::exception is caught
TBTC_Forward_a7fe5a32.GhostProba...   ERROR failure in update handler of 'Root': ParamFileSvc
ToolSvc                               ERROR Error initializing tool 'TBTC_Forward_a7fe5a32.GhostProbabilityTool_Long'
HiveDataBrokerSvc                     FATAL in sysInitialize(): exception with tag=TBTC_Forward_a7fe5a32 is caught
HiveDataBrokerSvc                     ERROR TBTC_Forward_a7fe5a32 	 Failed to retrieve tool GhostProbabilityTool_Long/GhostProbabilityTool_Long	 StatusCode=FAILURE

I've finally managed to build everything in debug mode and reproduce the issue :-) Then I've used valgrind, and after some hours, I found this :

==489323== Invalid write of size 4
==489323==    at 0xCD5A7277: DeUTSector::beetleStatus() const (DeUTSector.h:806)
==489323==    by 0xCD5B24AC: DeUTSector::initialize() (DeUTSector.cpp:281)
==489323==    by 0xACC1C5A5: XmlBaseDetElemCnv::fillObjRefs(IOpaqueAddress*, DataObject*) (XmlBaseDetElemCnv.cpp:153)
...
==489323==  Address 0xd2c895a0 is 0 bytes after a block of size 16 alloc'd
==489323==    at 0x484C617: operator new[](unsigned long) (vg_replace_malloc.c:725)
==489323==    by 0xCD5A70DD: std::__new_allocator<DeUTSector::Status>::allocate(unsigned long, void const*) (new_allocator.h:147)
==489323==    by 0xCD5A7180: allocate (allocator.h:198)
==489323==    by 0xCD5A7180: allocate (alloc_traits.h:482)
==489323==    by 0xCD5A7180: _M_allocate (stl_vector.h:378)
==489323==    by 0xCD5A7180: std::vector<DeUTSector::Status, std::allocator<DeUTSector::Status> >::_M_default_append(unsigned long) (vector.tcc:663)
==489323==    by 0xCD5A7220: std::vector<DeUTSector::Status, std::allocator<DeUTSector::Status> >::resize(unsigned long) (stl_vector.h:1011)
==489323==    by 0xCD5A7252: DeUTSector::beetleStatus() const (DeUTSector.h:800)

I've traced it to a recent change and commented that it should be fixed here : LHCb!4639 (comment 9363945)

I've now made a quick and dirty fix (overallocating by hand) and relaunched. I can confirm the problem seems to be gone.

mentioned in merge request LHCb!4639 (merged)

mentioned in merge request LHCb!5016 (merged)

closed with merge request LHCb!5016 (merged)

mentioned in commit LHCb@61c231d3

mentioned in commit LHCb@48bf9a69

mentioned in commit LHCb@cbdc7ab9