Hlt1SMOG2KsTopipi crashes tupling
When tupling is enabled for the line, running over certain events result in a memory crash. I haven't been able to debug it further than that but it's only when this particular line has tupling turned on. Files it can be reproduced with are /eos/lhcb/wg/rta/WP3/bandwidth_division/COLLISION25-run321834/bu_321834_LHCb_ECEB04_BU_0.mep,/eos/lhcb/wg/rta/WP3/bandwidth_division/COLLISION25-run321834/bu_321834_LHCb_ECEB04_BU_1.mep,/eos/lhcb/wg/rta/WP3/bandwidth_division/COLLISION25-run321834/bu_321834_LHCb_ECEB05_BU_0.mep,/eos/lhcb/wg/rta/WP3/bandwidth_division/COLLISION25-run321834/bu_321834_LHCb_ECEB05_BU_1.mep,/eos/lhcb/wg/rta/WP3/bandwidth_division/COLLISION25-run321834/bu_321834_LHCb_ECEB06_BU_0.mep,/eos/lhcb/wg/rta/WP3/bandwidth_division/COLLISION25-run321834/bu_321834_LHCb_ECEB06_BU_1.mep, somewhere in the last 80k events.
You can reproduce by running with the hlt1_pp_forward_then_matching_and_downstream_with_parkf_no_ghostkiller_tuning sequence and doing
$STACK/MooreOnline/build.${ARCHITECTURE}/run python \
$STACK/Allen/Dumpers/BinaryDumpers/options/allen.py \
--real-data \
--sequence=$STACK/Allen/InstallArea/${ARCHITECTURE}/constants/$seq.json \
--output-file=/tmp/hlt1_filtered$slice.mdf \
--mep $meps \
--tags $geometry,master \
--monitoring-filename /tmp/allen$slice.root | tee /tmp/log$slice
python3 /afs/cern.ch/work/a/aiwieder/private/RTA/BW/hlt1-bandwidth-division-tool/tuple_preparation/parse_log.py /tmp/log$slice /tmp/allen$slice.json The crash looks like
[FATAL] Process: 'P2800269' (SignalHandler) RTL:Handled signal: 11 [SIGSEGV] Old action:(nil) Mem:(nil) Code:00000080
[INFO] Process: 'P2800269' (ExitSignalHandler) ---------------------- Backtrace ----------------------
[INFO] Process: 'P2800269' Number of elements in backtrace: 13
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Online/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libOnlineBase.so.7.28.0.0(+0x13dda0)[0x7f37b1c94da0]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Online/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libOnlineBase.so.7.28.0.0(_ZN3RTL17ExitSignalHandler7handlerEiP9siginfo_tPv+0x304)[0x7f37b1c99804]
/lib64/libc.so.6(+0x3ebf0)[0x7f37da63ebf0]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_ZN24single_high_pt_muon_line26single_high_pt_muon_line_t6selectERKNS_10ParametersERKNS0_16DevicePropertiesESt5tupleIJKN5Allen5Views7Physics13BasicParticleEEE+0x48)[0x7f37b702de18]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_Z12process_lineIN24single_high_pt_muon_line26single_high_pt_muon_line_tENS0_10ParametersEEvPcPjS4_PK8LineDataPKSt5arrayIjLm10EEPKjjjjj+0x194)[0x7f37b702edc4]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_ZN17gather_selections9run_linesENS_10ParametersEjj+0x111)[0x7f37b7057fe1]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_ZNK17gather_selections19gather_selections_tclERKN5Allen5Store8StoreRefISt5tupleIJNS_10Parameters23host_number_of_events_tENS5_31host_selections_lines_offsets_tENS5_25host_selections_offsets_tENS5_29host_number_of_active_lines_tENS5_28host_names_of_active_lines_tENS5_22host_input_line_data_tENS5_24host_fn_parameters_agg_tENS5_19dev_fn_parameters_tENS5_28host_fn_parameter_pointers_tENS5_27dev_fn_parameter_pointers_tENS5_17host_fn_indices_tENS5_16dev_fn_indices_tENS5_20host_fn_parameters_tENS5_15dev_odin_data_tENS5_16dev_selections_tENS5_30dev_selections_lines_offsets_tENS5_24dev_selections_offsets_tENS5_28dev_number_of_active_lines_tENS5_16host_line_data_tENS5_15dev_line_data_tENS5_27dev_pre_scale_event_lists_tENS5_32dev_pre_scale_event_lists_size_tENS5_25dev_particle_containers_tENS5_29host_event_list_output_size_tENS5_28dev_event_list_output_size_tENS5_23dev_event_list_output_tEEES4_IJS6_S7_S8_S9_SA_SD_SE_SF_SG_SH_SI_SJ_SK_SL_SM_SN_SO_SP_SQ_SR_SS_ST_SU_SV_EES5_S4_IJSB_SC_EEEERK14RuntimeOptionsRK9ConstantsRKNS1_7ContextE+0x89c)[0x7f37b705d7ec]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_ZN6Stream3runEjRK14RuntimeOptions+0x646)[0x7f37b70b6806]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_Z10run_streammmiP6StreamSt10shared_ptrI14IInputProviderEP10IZeroMQSvcP14CheckerInvokerP11ROOTServicejbjb+0x75c)[0x7f37b6bc757c]
/afs/cern.ch/work/a/aiwieder/private/RTA/BW/stack-dev/Allen/InstallArea/x86_64_v2-el9-gcc13-opt/lib/libAllenLib.so(_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJPFvmmiP6StreamSt10shared_ptrI14IInputProviderEP10IZeroMQSvcP14CheckerInvokerP11ROOTServicejbjbEjjiS4_S7_S9_SB_SD_jbjbEEEEE6_M_runEv+0x55)[0x7f37b6b95da5]
/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9/lib64/libstdc++.so.6(+0xe04d3)[0x7f37d9ce04d3]
/lib64/libc.so.6(+0x8a0ea)[0x7f37da68a0ea]
/lib64/libc.so.6(+0x10f150)[0x7f37da70f150]
[INFO] Process: 'P2800269' (SignalHandler) 00 --> 0x7f37b1c94da0
[INFO] Process: 'P2800269' (SignalHandler) 01 --> 0x7f37b1c99804
[INFO] Process: 'P2800269' (SignalHandler) 02 --> 0x7f37da63ebf0
[INFO] Process: 'P2800269' (SignalHandler) 03 --> 0x7f37b702de18
[INFO] Process: 'P2800269' (SignalHandler) 04 --> 0x7f37b702edc4
[INFO] Process: 'P2800269' (SignalHandler) 05 --> 0x7f37b7057fe1
[INFO] Process: 'P2800269' (SignalHandler) 06 --> 0x7f37b705d7ec
[INFO] Process: 'P2800269' (SignalHandler) 07 --> 0x7f37b70b6806
[INFO] Process: 'P2800269' (SignalHandler) 08 --> 0x7f37b6bc757c
[INFO] Process: 'P2800269' (SignalHandler) 09 --> 0x7f37b6b95da5
[INFO] Process: 'P2800269' (SignalHandler) 10 --> 0x7f37d9ce04d3
[INFO] Process: 'P2800269' (SignalHandler) 11 --> 0x7f37da68a0ea
[INFO] Process: 'P2800269' (SignalHandler) 12 --> 0x7f37da70f150Not that even though the trace sometimes mentions a certain line it is unrelated, it does that even if that line doesn't have tupling turned on. I don't know why though.