test_lbexec_hlt2 test_lbexec_hlt2_pp_commissioning timeouts during finalize
test_lbexec_hlt2 and test_lbexec_hlt2_pp_commissioning frequently timeout and at least in two occasions the stack trace when the process gets terminated points to the Gaudi monitoring code.
For example, see here
relevant stack trace:
Thread 1 (Thread 0x7f1fd2d09740 (LWP 13161) "python"):
#0 0x00007f1f91e9e018 in std::__find_if<std::_Deque_iterator<Gaudi::Monitoring::Hub::Entity, Gaudi::Monitoring::Hub::Entity&, Gaudi::Monitoring::Hub::Entity*>, __gnu_cxx::__ops::_Iter_equals_val<Gaudi::Monitoring::Hub::Entity const> > (__first=..., __last=..., __pred=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algobase.h:2077
#1 0x00007f1f91e9e214 in std::__find_if<std::_Deque_iterator<Gaudi::Monitoring::Hub::Entity, Gaudi::Monitoring::Hub::Entity&, Gaudi::Monitoring::Hub::Entity*>, __gnu_cxx::__ops::_Iter_equals_val<Gaudi::Monitoring::Hub::Entity const> > (__first=..., __last=..., __pred=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algobase.h:2113
#2 0x00007f1f91e9e277 in std::find<std::_Deque_iterator<Gaudi::Monitoring::Hub::Entity, Gaudi::Monitoring::Hub::Entity&, Gaudi::Monitoring::Hub::Entity*>, Gaudi::Monitoring::Hub::Entity> (__first=..., __last=..., __val=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algo.h:3852
#3 0x00007f1f91ea1360 in Gaudi::Monitoring::BaseSink::removeEntity (this=0x258b49f0, ent=...) at ../GaudiKernel/include/Gaudi/BaseSink.h:47
#4 0x00007f1f924566c9 in Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> >(Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>&)::{lambda(auto:1)#1}::operator()<Gaudi::Monitoring::Hub::Sink*>(Gaudi::Monitoring::Hub::Sink*) const (sink=<optimized out>, __closure=<synthetic pointer>) at /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12+detdesc-dbg/include/Gaudi/MonitoringHub.h:174
#5 std::for_each<std::_Deque_iterator<Gaudi::Monitoring::Hub::Sink*, Gaudi::Monitoring::Hub::Sink*&, Gaudi::Monitoring::Hub::Sink**>, Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> >(Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>&)::{lambda(auto:1)#1}>(std::_Deque_iterator<Gaudi::Monitoring::Hub::Sink*, Gaudi::Monitoring::Hub::Sink*&, Gaudi::Monitoring::Hub::Sink**>, std::_Deque_iterator<Gaudi::Monitoring::Hub::Sink*, Gaudi::Monitoring::Hub::Sink*&, Gaudi::Monitoring::Hub::Sink**>, Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> >(Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>&)::{lambda(auto:1)#1}) (__first=..., __last=..., __f=__f@entry=...) at /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/12.1.0-57c96/x86_64-centos7/include/c++/12.1.0/bits/stl_algo.h:3787
#6 0x00007f1f92458300 in Gaudi::Monitoring::Hub::removeEntity<Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1> > (this=0x24cbaa88, ent=...) at /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12+detdesc-dbg/include/Gaudi/MonitoringHub.h:174
#7 0x00007f1f92458363 in Gaudi::Accumulators::MsgCounter<(MSG::Level)4, (Gaudi::Accumulators::atomicity)1>::~MsgCounter (this=0x51c823f8, __in_chrg=<optimized out>) at /workspace/build/Gaudi/InstallArea/x86_64_v2-centos7-gcc12+detdesc-dbg/include/Gaudi/Accumulators.h:1147
#8 0x00007f1f77b9ace7 in LoKi::DistanceCalculator::~DistanceCalculator (this=this@entry=0x51c80310, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at ../Phys/LoKiFitters/src/DistanceCalculator.h:52
#9 0x00007f1f77b9af5f in LoKi::DistanceCalculator::~DistanceCalculator (this=0x51c80310, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at ../Phys/LoKiFitters/src/DistanceCalculator.h:52
#10 0x00007f1f9531fd9d in implements<IAlgTool, IDataHandleHolder, IProperty, IStateful>::release (this=<optimized out>) at ../GaudiKernel/include/GaudiKernel/implements.h:53
#11 0x00007f1f91de1543 in ToolSvc::finalize (this=0x1ffc18b0) at ../GaudiCoreSvc/src/ApplicationMgr/ToolSvc.cpp:185
#12 0x00007f1f9543f0b5 in Service::sysFinalize (this=0x1ffc18b0) at ../GaudiKernel/src/Lib/Service.cpp:202
#13 0x00007f1f91ddba67 in ServiceManager::finalize (this=0x24cbe480) at ../GaudiCoreSvc/src/ApplicationMgr/ServiceManager.cpp:451
#14 0x00007f1f91dbda74 in ApplicationMgr::finalize (this=0x24cba260) at ../GaudiCoreSvc/src/ApplicationMgr/ApplicationMgr.cpp:612
#15 0x00007f1f95326d0d in Gaudi::Application::run (this=0x1f86f920) at ../GaudiKernel/src/Lib/Application.cpp:91last lines in stdout:
SelectiveCombineRawBankViewsToRawEvent/SelectiveCombineRawBanks_for_default #=19 Sum=19 Eff=|( 100.0000 +- 0.00000 )%|
AddressKillerAlg/AddressKillerAlg_5d7e77b6 #=19 Sum=19 Eff=|( 100.0000 +- 0.00000 )%|
RawEventSimpleCombiner/RawEventSimpleCombiner_9a71b658 #=19 Sum=19 Eff=|( 100.0000 +- 0.00000 )%|
CopyInputStream/CopyInputStream_db8c480d #=19 Sum=19 Eff=|( 100.0000 +- 0.00000 )%|
HLTControlFlowMgr INFO Histograms converted successfully according to request.
TransportSvc SUCCESS GEOMETRY ERRORS: 'Skip' map has the size 0
TransportSvc SUCCESS GEOMETRY ERRORS: 'Recover' map has the size 0
TransportSvc SUCCESS GEOMETRY ERRORS: 'Codes' map has the size 0
TransportSvc INFO Reset the static pointer to DetDesc::IGeometyrErrorSvc
ToolSvc INFO Removing all tools created by ToolSvc
FunctionalDiElectronMaker_2d010d... SUCCESS Exceptions/Errors/Warnings/Infos Statistics : 0/0/5/0
FunctionalDiElectronMaker_2d010d... SUCCESS #WARNINGS = 1 Message = 'No convergency has been reached'
FunctionalDiElectronMaker_2d010d... SUCCESS #WARNINGS = 1 Message = '_iterate_opt: No convergency has been reached'
FunctionalDiElectronMaker_2d010d... SUCCESS #WARNINGS = 3 Message = 'fit(): Vertex is outside of fiducial volume'
FunctionalDiElectronMaker_2d010d... SUCCESS #WARNINGS = 1 Message = 'fit(): failure from _iterate()'
FunctionalDiElectronMaker_2d010d... SUCCESS #WARNINGS = 1 Message = 'fit(): failure from fit'@sponce could you please take a look if there is a real problem?