Crash when running chained HLT1 and HLT2 with MooreAnalysis

Report of a crash/segfault with Moore

Brief summary of encountered problem

I am encountering a crash when running chained HLT1 and HLT2 selection with MooreAnalysis to calculate HLT2 efficiencies. out_hlt1_hlt2.txt

Dataset/Options file you were running on

The scripts that were used to run the HLT selection including links to the dst files are linked below. hlt2_lines.py hlt2_eff_11114002.py

Moore version used (or nightly, as appropriate)

In order to reproduce this issue the linked Moore and ParamsFile branches have to be used.

Additional information

added Crash label

assigned to @lecarus

changed the description

assigned to @gligorov

assigned to @dovombru

@msaur

mentioned in merge request !1224 (merged)

Similar crash observed when running Allen monitoring using 100 000 events:

Moore/run gaudirun.py DBASE/PRConfig/options/Moore/DataChallenges/DC_Sim10aU1_MinBias.py DBASE/PRConfig/options/Moore/DataChallenges/DC_Sim10aU1_default_conds_100k.py Moore/Hlt/Moore/tests/options/DC_histo_file.py Moore/Hlt/RecoConf/options/hlt1_reco_allen_trackresolution.py

where hlt1_reco_allen_trackresolution is taken from 6b4706a0

This crash is observed only when processing larger number of events, a simple test of 1000 events finished fine,Allen_monitoring_crash_20220224_hlt1_reco_allen_trackresolution.txt

Possibly the most relevant parts:

#7  0x00007fc0fb18b4c0 in AllenForwardToV2Tracks::operator()(HostBuffers const&) const () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenWrapper.so
#8  0x00007fc0fb19b7ba in Gaudi::Functional::details::MultiTransformer<std::tuple<std::vector<LHCb::Event::v2::Track, std::allocator<LHCb::Event::v2::Track> >, std::vector<LHCb::Event::v2::Track, std::allocator<LHCb::Event::v2::Track> > > (HostBuffers const&), Gaudi::Functional::Traits::BaseClass_t<GaudiHistoAlg>, true>::execute() () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenWrapper.so
#9  0x00007fc1066f7310 in Gaudi::Algorithm::sysExecute(EventContext const&) () from /interactive_storage/msaur/master_stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libGaudiKernel.so
#10 0x00007fc10510fdc0 in GaudiAlgorithm::sysExecute(EventContext const&) () from /interactive_storage/msaur/master_stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libGaudiAlgLib.so
#11 0x00007fc1007da5dc in AlgWrapper::execute(EventContext&, gsl::span<LHCb::Interfaces::ISchedulerConfiguration::State::AlgState, 18446744073709551615ul>) const () from /interactive_storage/msaur/master_stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libHLTScheduler.so
#12 0x00007fc1007cc35c in HLTControlFlowMgr::push(EventContext&&)::{lambda(EventContext&)#1}::operator()(EventContext&) const () from /interactive_storage/msaur/master_stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libHLTScheduler.so
#13 0x00007fc1007ccbd1 in tbb::internal::function_task<(anonymous namespace)::EventTask<HLTControlFlowMgr::push(EventContext&&)::{lambda(EventContext&)#1}> >::execute() () from /interactive_storage/msaur/master_stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libHLTScheduler.so
#14 0x00007fc106a8ba45 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this
entry=0x7fc104d83e00, context_guard=..., t=0x7fc104d8bb40, isolation=isolation
entry=0) at ../../src/tbb/custom_scheduler.h:474
#15 0x00007fc106a8bd78 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc104d83e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:636
#16 0x00007fc106a85857 in tbb::internal::arena::process (this=0x7fc104d93a00, s=...) at ../../src/tbb/arena.cpp:196
#17 0x00007fc106a84060 in tbb::internal::market::process (this=0x7fc104d9ab00, j=...) at ../../src/tbb/market.cpp:667
#18 0x00007fc106a807ac in tbb::internal::rml::private_worker::run (this=0x7fc0fd2a6180) at ../../src/tbb/private_server.cpp:266
#19 0x00007fc106a809e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#20 0x00007fc12b706ea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc12ad26b0d in clone () from /lib64/libc.so.6

Another, but possibly different crash when running same settings with just 10 000 events (full log attached): Allen_monitoring_crash_20220224_hlt1_reco_allen_trackresolution_10k.txt

#7  0x00007f7231a427e2 in Allen::TypeErasedAlgorithm::TypeErasedAlgorithm<low_pt_di_muon_line::low_pt_di_muon_line_t>(std::in_place_type_t<low_pt_di_muon_line::low_pt_di_muon_line_t>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)::{lambda(void*, std::any&, RuntimeOptions const&, Constants const&, HostBuffers const&)#3}::_FUN(void*, std::any&, RuntimeOptions const&, Constants const&, HostBuffers const&) () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenLib.so
#8  0x00007f7231817abf in Stream::run(unsigned int, RuntimeOptions const&) () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenLib.so
#9  0x00007f7231f6c555 in RunAllen::operator()(std::array<std::tuple<std::vector<char, std::allocator<char> >, int>, 87ul> const&, LHCb::ODINImplementation::v7::ODIN const&) const () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenWrapper.so
#10 0x00007f7231f7e731 in _ZZN5Gaudi10Functional7details22MultiTransformerFilterIFSt5tupleIJ11HostBuffersEERKSt5arrayIS3_IJSt6vectorIcSaIcEEiEELm87EERKN4LHCb18ODINImplementation2v74ODINEENS0_6Traits4use_IJEEELb1EE7executeEvENKUlDpRT_E_clIJ21DataObjectWriteHandleIS4_S4_EEEEDaSR_ () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenWrapper.so
#11 0x00007f7231f7ef91 in Gaudi::Functional::details::MultiTransformerFilter<std::tuple<HostBuffers> (std::array<std::tuple<std::vector<char, std::allocator<char> >, int>, 87ul> const&, LHCb::ODINImplementation::v7::ODIN const&), Gaudi::Functional::Traits::use_<>, true>::execute() () from /interactive_storage/msaur/master_stack/Allen/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libAllenWrapper.so
#12 0x00007f723d45f310 in Gaudi::Algorithm::sysExecute(EventContext const&) () from /interactive_storage/msaur/master_stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libGaudiKernel.so
#13 0x00007f723be77dc0 in GaudiAlgorithm::sysExecute(EventContext const&) () from /interactive_storage/msaur/master_stack/Gaudi/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libGaudiAlgLib.so
#14 0x00007f72375425dc in AlgWrapper::execute(EventContext&, gsl::span<LHCb::Interfaces::ISchedulerConfiguration::State::AlgState, 18446744073709551615ul>) const () from /interactive_storage/msaur/master_stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libHLTScheduler.so
#15 0x00007f723753435c in HLTControlFlowMgr::push(EventContext&&)::{lambda(EventContext&)#1}::operator()(EventContext&) const () from /interactive_storage/msaur/master_stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libHLTScheduler.so
#16 0x00007f7237534bd1 in tbb::internal::function_task<(anonymous namespace)::EventTask<HLTControlFlowMgr::push(EventContext&&)::{lambda(EventContext&)#1}> >::execute() () from /interactive_storage/msaur/master_stack/LHCb/InstallArea/x86_64_v2-centos7-gcc11-opt/lib/libHLTScheduler.so
#17 0x00007f723d7f3a45 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this
entry=0x7f723baebe00, context_guard=..., t=0x7f723baf3b40, isolation=isolation
entry=0) at ../../src/tbb/custom_scheduler.h:474
#18 0x00007f723d7f3d78 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f723baebe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:636
#19 0x00007f723d7ed857 in tbb::internal::arena::process (this=0x7f723bafba00, s=...) at ../../src/tbb/arena.cpp:196
#20 0x00007f723d7ec060 in tbb::internal::market::process (this=0x7f723bb02b00, j=...) at ../../src/tbb/market.cpp:667
#21 0x00007f723d7e87ac in tbb::internal::rml::private_worker::run (this=0x7f723400e180) at ../../src/tbb/private_server.cpp:266
#22 0x00007f723d7e89e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#23 0x00007f726246eea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f7261a8eb0d in clone () from /lib64/libc.so.6

mentioned in merge request !1350 (merged)

mentioned in issue MooreAnalysis#30 (closed)

Have there been new developments on this? I am asking because I've got an approaching deadline and it would be important to know if this will be fixed in time. @dovombru

Allen!792 (merged) is a fix for crashes observed in HLT1+HLT2 MooreAnalysis running. Can you please check whether this also fixes your problem?

I updated my stack and I immediately get an error complaining about KeyError: 'VPRetinaCluster'. I added the full output. new_error.txt @dovombru

I also tried to compile with make Allen CMAKEFLAGS="-DSEQUENCES=hlt1_pp_veloSP" and allen_sequence.bind(sequence="hlt1_pp_veloSP"). I still get the same error

Have you tried to apply Allen!795 (merged) ?

You probably need to purge the stack build directories and re-compile. This is likely due to the new raw event format version introduced with the Retina raw banks (see https://mattermost.web.cern.ch/lhcb/pl/fbi3oiaiebfu5faecyjry49bir)

I set up the stack completely new. I still get this error.

The retina clusters are now the default, but you're likely running on a file that doesn't have them. Either use a file that does, or use the hlt1_pp_veloSP sequence.

Changing the sequence can be done using:

from RecoConf.hlt1_allen import sequence
with sequence.bind(sequence="hlt1_pp_veloSP"):
    # whatever you were doing before

https://gitlab.cern.ch/lhcb/Allen/-/blob/master/doc/setup/input_files.rst describes where to find samples with Retina clusters and how to add them to your sample.

@dovombru I tried all three methods. Using an already existing file with retina clusters (/eos/lhcb/wg/rta/WP6/Allen/mdf_input/RetinaCluster_samples/Upgrade_KstEE_MU_FTv4_DIGI_retinacluster_digi/kstee_retinacluster.digi), adding retina clusters to one of my samples and using hlt1_pp_veloSP. I always get the same error. Its obviously not the point of this Issue to debug this, but I also can't check whether the previous crash still exists.

@lecarus I tested a fresh stack setup with the master branches of Moore and MooreAnalysis using the slightly modified options scripts below with the sample including Retina raw banks. Here I don't observe any errors.

hlt_eff_11114002.py

hlt2_lines.py

Please note that there has been one more fix by @gbassi for running on samples that do not include Retina raw anks: !1429 (merged) and related merge requests.

mentioned in merge request Allen!792 (merged)

closed