Moore memory usage too high in APs
The Moore memory usage is too high in some APs that run HLT1 and HLT2 jobs on MC samples before tupling, causing the jobs to be marked as failed. Some recent examples:
-
lhcb-datapkg/AnalysisProductions#625 (closed) reported by @lugrazet (discussion at https://mattermost.web.cern.ch/lhcb/pl/93n976ctnjrwjrx39jdduwjepw ): uses
Moore v55r7andMoore v55r10p1 -
lhcb-datapkg/AnalysisProductions!1523 (merged) reported by @smaccoli (discussion at https://mattermost.web.cern.ch/lhcb/pl/doazciq7wjyybdubeqwj6f1fqe ): uses
Moore v55r11
@lugrazet and @cburr tried modifying the basket size and AgeLimit, which helps a bit but is not sufficient in all cases (details in lhcb-datapkg/AnalysisProductions#625 (closed) ). Update: AgeLimit=0 is now the default in all v55r* releases.
We have seen issues with memory usage in the past, with related issues:
-
MooreOnline#77 running HLT2 on real data. Two fixes were implemented (Rec!3941 (merged) and Rec!3952 (merged)) and are included in
Moore >= v55r10 - #529 (closed): running HLT1 and HLT2 on 2022 MC productions. The main fix was gaudi/Gaudi!1381 (merged) Is this still being used @cburr ? The usage of many monitoring algorithms which are probably not useful for MC productions was also pointed out. Do we still run with them @jonrob @sesen ?
FYI @cagapopo @msaur fell free to reassign cc @cburr @nskidmor @decianm @mfontana please add any further information
Edited by Chris Burr