Skip to content

test_lbexec_hlt2 test_lbexec_hlt2_pp_commissioning timeouts during finalize

test_lbexec_hlt2 and test_lbexec_hlt2_pp_commissioning frequently timeout and at least in two occasions the stack trace when the process gets terminated points to the Gaudi monitoring code.

For example, see here

relevant stack trace:

Thread 1 (Thread 0x7f1fd2d09740 (LWP 13161) "python"):
#0  0x00007f1f91e9e018 in std::__find_if<std::_Deque_iterator<Gaudi::Monitoring::Hub::Entity, Gaudi::Monitoring::Hub::Entity&, Gaudi::Monitoring::Hub::Entity*>, __gnu_cxx::__ops::_Iter_equals_val<Gaudi::Monitoring::Hub::Entity const> > (__first=..., __last=..., __pred=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algobase.h:2077
#1  0x00007f1f91e9e214 in std::__find_if<std::_Deque_iterator<Gaudi::Monitoring::Hub::Entity, Gaudi::Monitoring::Hub::Entity&, Gaudi::Monitoring::Hub::Entity*>, __gnu_cxx::__ops::_Iter_equals_val<Gaudi::Monitoring::Hub::Entity const> > (__first=..., __last=..., __pred=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algobase.h:2113
#2  0x00007f1f91e9e277 in std::find<std::_Deque_iterator<Gaudi::Monitoring::Hub::Entity, Gaudi::Monitoring::Hub::Entity&, Gaudi::Monitoring::Hub::Entity*>, Gaudi::Monitoring::Hub::Entity> (__first=..., __last=..., __val=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algo.h:3852
#3  0x00007f1f91ea1360 in Gaudi::Monitoring::BaseSink::removeEntity (this=0x258b49f0, ent=...) at ../GaudiKernel/include/Gaudi/BaseSink.h:47
#4  0x00007f1f924566c9 in Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> >(Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>&)::{lambda(auto:1)#1}::operator()<Gaudi::Monitoring::Hub::Sink*>(Gaudi::Monitoring::Hub::Sink*) const (sink=<optimized out>, __closure=<synthetic pointer>) at /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12+detdesc-dbg/include/Gaudi/MonitoringHub.h:174
#5  std::for_each<std::_Deque_iterator<Gaudi::Monitoring::Hub::Sink*, Gaudi::Monitoring::Hub::Sink*&, Gaudi::Monitoring::Hub::Sink**>, Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> >(Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>&)::{lambda(auto:1)#1}>(std::_Deque_iterator<Gaudi::Monitoring::Hub::Sink*, Gaudi::Monitoring::Hub::Sink*&, Gaudi::Monitoring::Hub::Sink**>, std::_Deque_iterator<Gaudi::Monitoring::Hub::Sink*, Gaudi::Monitoring::Hub::Sink*&, Gaudi::Monitoring::Hub::Sink**>, Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> >(Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>&)::{lambda(auto:1)#1}) (__first=..., __last=..., __f=__f@entry=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algo.h:3787
#6  0x00007f1f92458300 in Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> > (this=0x24cbaa88, ent=...) at /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12+detdesc-dbg/include/Gaudi/MonitoringHub.h:174
#7  0x00007f1f92458363 in Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>::~MsgCounter (this=0x51c823f8, __in_chrg=<optimized out>) at /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12+detdesc-dbg/include/Gaudi/Accumulators.h:1147
#8  0x00007f1f77b9ace7 in LoKi::DistanceCalculator::~DistanceCalculator (this=this@entry=0x51c80310, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at ../Phys/LoKiFitters/src/DistanceCalculator.h:52
#9  0x00007f1f77b9af5f in LoKi::DistanceCalculator::~DistanceCalculator (this=0x51c80310, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at ../Phys/LoKiFitters/src/DistanceCalculator.h:52
#10 0x00007f1f9531fd9d in implements<IAlgTool, IDataHandleHolder, IProperty, IStateful>::release (this=<optimized out>) at ../GaudiKernel/include/GaudiKernel/implements.h:53
#11 0x00007f1f91de1543 in ToolSvc::finalize (this=0x1ffc18b0) at ../GaudiCoreSvc/src/ApplicationMgr/ToolSvc.cpp:185
#12 0x00007f1f9543f0b5 in Service::sysFinalize (this=0x1ffc18b0) at ../GaudiKernel/src/Lib/Service.cpp:202
#13 0x00007f1f91ddba67 in ServiceManager::finalize (this=0x24cbe480) at ../GaudiCoreSvc/src/ApplicationMgr/ServiceManager.cpp:451
#14 0x00007f1f91dbda74 in ApplicationMgr::finalize (this=0x24cba260) at ../GaudiCoreSvc/src/ApplicationMgr/ApplicationMgr.cpp:612
#15 0x00007f1f95326d0d in Gaudi::Application::run (this=0x1f86f920) at ../GaudiKernel/src/Lib/Application.cpp:91

last lines in stdout:

   SelectiveCombineRawBankViewsToRawEvent/SelectiveCombineRawBanks_for_default                                         #=19      Sum=19          Eff=|( 100.0000 +- 0.00000 )%|
   AddressKillerAlg/AddressKillerAlg_5d7e77b6                                                                          #=19      Sum=19          Eff=|( 100.0000 +- 0.00000 )%|
   RawEventSimpleCombiner/RawEventSimpleCombiner_9a71b658                                                              #=19      Sum=19          Eff=|( 100.0000 +- 0.00000 )%|
   CopyInputStream/CopyInputStream_db8c480d                                                                            #=19      Sum=19          Eff=|( 100.0000 +- 0.00000 )%|

HLTControlFlowMgr                      INFO Histograms converted successfully according to request.
TransportSvc                        SUCCESS  GEOMETRY ERRORS: 'Skip'     map has the size 0

TransportSvc                        SUCCESS  GEOMETRY ERRORS: 'Recover'  map has the size 0

TransportSvc                        SUCCESS  GEOMETRY ERRORS: 'Codes'    map has the size 0

TransportSvc                           INFO Reset the static pointer to DetDesc::IGeometyrErrorSvc
ToolSvc                                INFO Removing all tools created by ToolSvc
FunctionalDiElectronMaker_2d010d... SUCCESS Exceptions/Errors/Warnings/Infos Statistics : 0/0/5/0
FunctionalDiElectronMaker_2d010d... SUCCESS  #WARNINGS   = 1        Message = 'No convergency has been reached'
FunctionalDiElectronMaker_2d010d... SUCCESS  #WARNINGS   = 1        Message = '_iterate_opt: No convergency has been reached'
FunctionalDiElectronMaker_2d010d... SUCCESS  #WARNINGS   = 3        Message = 'fit(): Vertex is outside of fiducial volume'
FunctionalDiElectronMaker_2d010d... SUCCESS  #WARNINGS   = 1        Message = 'fit(): failure from _iterate()'
FunctionalDiElectronMaker_2d010d... SUCCESS  #WARNINGS   = 1        Message = 'fit(): failure from fit'

@sponce could you please take a look if there is a real problem?