High memory usage in MC productions
As has been discussed in a few places, the recent 2022 MC productions have been unable to run due to the memory usage being too high. I've included both vmem and rss however rss is the only one which actually matters.
Plots of memory usage over time
HLT1
HLT2
Reproducers
I've exactracted a random MCReconstruction job to make a standalone reproducer that can be used for debugging.
HLT1 reproducer
xrdcp root://eoslhcb.cern.ch//eos/lhcb/user/c/cburr/2022-mc-reco-mem-debug/00175875_00005299_1.xdigi .
xrdcp root://eoslhcb.cern.ch//eos/lhcb/user/c/cburr/2022-mc-reco-mem-debug/prodConf_Moore_00175875_00005299_2.json .
lb-dirac lb-prod-run --verbose --interactive prodConf_Moore_00175875_00005299_2.json
# At this point you can run the job with (or exit the shell and use your own Moore build)
# lbexec Moore.production:allen_hlt1_production lbexec_options_00175875_00005299_2.yaml hlt1_pp_no_gec_no_utHLT2 reproducer
xrdcp root://eoslhcb.cern.ch//eos/lhcb/user/c/cburr/2022-mc-reco-mem-debug/00175875_00005299_2.xdst .
xrdcp root://eoslhcb.cern.ch//eos/lhcb/user/c/cburr/2022-mc-reco-mem-debug/prodConf_Moore_00175875_00005299_3.json .
lb-dirac lb-prod-run --verbose --interactive prodConf_Moore_00175875_00005299_3.json
# At this point you can run the job with (or exit the shell and use your own Moore build)
# lbexec Moore.production:hlt2_pp_commissioning lbexec_options_00175875_00005299_3.yamlcc @romanov