Compilation of Stream.cpp needs too much resources
The compilation of https://gitlab.cern.ch/lhcb/Allen/-/blob/a67675aced52d788ecba6877ea1f2cd2e5d67488/stream/sequence/src/Stream.cpp needs too much memory, especially for dbg
builds. The practical consequence is that the nightly build VMs fill up their memory (~14 GB, but some tests might run concurrently), start swapping and the compilation does not advance. This results in the build jobs being stuck and killed, needing to restart from scratch (from Gaudi). Ultimately, since we need to check dbg
slots for accepting MRs this slows down the feedback. (Thankfully, now there is not much MR traffic, but that will soon change).
We should find a fix for this or if that cannot be done quickly we need a workaround.
To illustrate, at 11:40 (1.5 hours after the compilation started) the process looks like this. The machine is otherwise not doing much at all (so even if a test slowed down the compilation initially, that is not the case now).
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
lblocal 32644 32640 15 2553804 7588396 7 10:08 ? D 13:39 /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/../libexec/gcc/x86_64-pc-linux-gnu/9.2.0/cc1plus -quiet -I /workspace/build/Allen/build/stream -I /workspace/build/Allen/stream -I /workspace/build/Allen/build/configuration/sequences -I /workspace/build/Allen/external -I /workspace/build/Allen/backend/include -I /workspace/build/Allen/mdf/include -I /workspace/build/Allen/device/selections/Postscale/include -I /workspace/build/Allen/device/selections/lines/include -I /workspace/build/Allen/device/selections/line_types/include -I /workspace/build/Allen/device/selections/Hlt1/include -I /workspace/build/Allen/device/vertex_fit/vertex_fitter/include -I /workspace/build/Allen/device/vertex_fit/common/include -I /workspace/build/Allen/device/kalman/ParKalman/include -I /workspace/build/Allen/device/PV/beamlinePV/include -I /workspace/build/Allen/checker/selections/include -I /workspace/build/Allen/checker/pv/include -I /workspace/build/Allen/checker/tracking/include -I /workspace/build/Allen/host/init_event_list/include -I /workspace/build/Allen/host/prefix_sum/include -I /workspace/build/Allen/host/global_event_cut/include -I /workspace/build/Allen/host/utils/prefix_sum/include -I /workspace/build/Allen/host/velo/clustering/include -I /workspace/build/Allen/device/raw_banks/include -I /workspace/build/Allen/device/PV/common/include -I /workspace/build/Allen/device/PV/patPV/include -I /workspace/build/Allen/device/utils/include -I /workspace/build/Allen/device/muon/match_upstream_muon/include -I /workspace/build/Allen/device/muon/decoding/include -I /workspace/build/Allen/device/muon/classification/include -I /workspace/build/Allen/device/muon/is_muon/include -I /workspace/build/Allen/device/muon/preprocessing/include -I /workspace/build/Allen/device/muon/common/include -I /workspace/build/Allen/device/associate/include -I /workspace/build/Allen/device/SciFi/consolidate/include -I /workspace/build/Allen/device/SciFi/looking_forward/include -I /workspace/build/Allen/device/SciFi/common/include -I /workspace/build/Allen/device/SciFi/preprocessing/include -I /workspace/build/Allen/device/velo/simplified_kalman_filter/include -I /workspace/build/Allen/device/velo/search_by_triplet/include -I /workspace/build/Allen/device/velo/prefix_sum/include -I
/workspace/build/Allen/device/velo/mask_clustering/include -I /workspace/build/Allen/device/velo/consolidate_tracks/include -I /workspace/build/Allen/device/velo/calculate_phi_and_sort/include -I /workspace/build/Allen/device/velo/common/include -I /workspace/build/Allen/device/UT/compassUT/include -I /workspace/build/Allen/device/UT/consolidate/include -I /workspace/build/Allen/device/UT/sorting/include -I /workspace/build/Allen/device/UT/UTDecoding/include -I /workspace/build/Allen/device/UT/common/include -I /workspace/build/Allen/device/event_model/associate/include -I /workspace/build/Allen/device/event_model/muon/include -I /workspace/build/Allen/device/event_model/SciFi/include -I /workspace/build/Allen/device/event_model/UT/include -I /workspace/build/Allen/device/event_model/common/include -I /workspace/build/Allen/device/event_model/velo/include -I /workspace/build/Allen/device/utils/binary_search/include -I /workspace/build/Allen/device/utils/sorting/include -I /workspace/build/Allen/device/utils/float_operations/include -I /workspace/build/Allen/device/utils/prefix_sum/include -I /workspace/build/Allen/main/include -I /workspace/build/Allen/stream/checkers/include -I /workspace/build/Allen/stream/setup/include -I /workspace/build/Allen/stream/sequence/include -I /workspace/build/Allen/stream/scheduler/include -I /workspace/build/Allen/stream/memory_manager/include -I /workspace/build/Allen/stream/gear/include -I /workspace/build/Allen/test/contracts/include -I /workspace/build/Allen/build/include -I /workspace/build/Rec/InstallArea/x86_64-centos7-gcc9+py3-dbg/include -I /workspace/build/LHCb/InstallArea/x86_64-centos7-gcc9+py3-dbg/include -iprefix /cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/../lib/gcc/x86_64-pc-linux-gnu/9.2.0/ -D_GNU_SOURCE -D_REENTRANT -D BOOST_ALL_NO_LIB -D BOOST_FILESYSTEM_DYN_LINK -D BOOST_FILESYSTEM_VERSION=3 -D BOOST_REGEX_DYN_LINK -D BOOST_SPIRIT_USE_PHOENIX_V3 -D BOOST_SYSTEM_DYN_LINK -D BOOST_THREAD_DYN_LINK -D FMT_LOCALE -D FMT_SHARED -D GAUDI_V20_COMPAT -D GOD_NOALLOC
-D MEMORY_MANAGER_SINGLE_ALLOC -D ROOT_CXX17 -D TARGET_DEVICE_CPU -D WITH_ROOT -D _GNU_SOURCE -D f2cFortran -D linux -D unix -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_97apython3/ROOT/v6.20.06/x86_64-centos7-gcc9-dbg/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/Boost/1.72.0-30ed7/x86_64-centos7-gcc9-dbg/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_97apython3/cppgsl/3.0.1/x86_64-centos7-gcc9-dbg/include -isystem
/workspace/build/Gaudi/InstallArea/x86_64-centos7-gcc9+py3-dbg/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_97apython3/tbb/2020_U1/x86_64-centos7-gcc9-dbg/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_97apython3/fmt/6.1.2/x86_64-centos7-gcc9-dbg/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/LCG_97apython3/jsonmcpp/3.6.1/x86_64-centos7-gcc9-dbg/include /workspace/build/Allen/stream/sequence/src/Stream.cpp -quiet -dumpbase Stream.cpp -march=x86-64 -msse4.2 -auxbase-strip CMakeFiles/Stream.dir/sequence/src/Stream.cpp.o -g -Og -Wall -Wextra -Werror=return-type -Wpedantic -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -Wsuggest-override -std=c++17 -std=c++17 -fdiagnostics-color=always -fmessage-length=0 -fPIC -o -