Observed rate drops to zero for sprucing line - two unexpected features

I was trying to understand why some sprucing lines that have been recently merged and that I tested locally to have the rate of <0.5kHz, are now at 0kHz in the tests. I was able to boil down this discrepancy to two main observations. (Notice that I performed these tests with a fresh new stack). To reproduce please checkout the following Moore branch: https://gitlab.cern.ch/lhcb/Moore/-/tree/matzeni/strange-behavior-sprucing-rate-issue-605 and recompile Moore (I did make fast/Moore).

Observation N.1

If I run the script

run gaudirun.py Hlt/Hlt2Conf/tests/options/bandwidth/spruce_bandwidth_input.py Hlt/Hlt2Conf/tests/options/bandwidth/spruce_bandwidth_wg_streams_smalltest.py

(where the latter is a variation on spruce_bandwidth_wg_streams.pyhere where I only keep the rd lines) over 10K events I obtain the following number of candidates for the lines of interest:

"Monitor_SpruceRD_BuToHpEE_SameSign_n_candidates": 0
"Monitor_SpruceRD_BuToHpEE_n_candidates": 0
"Monitor_SpruceRD_BuToHpMuMu_SameSign_n_candidates": 0
"Monitor_SpruceRD_BuToHpMuMu_n_candidates": 0
"Monitor_SpruceRD_B0ToHpHmEE_SameSign_n_candidates": 0
"Monitor_SpruceRD_B0ToHpHmMuMu_n_candidates": 0
"Monitor_SpruceRD_B0ToHpHmMuMu_SameSign_n_candidates": 0
"Monitor_SpruceRD_B0ToHpHmEE_n_candidates": 0

This corresponds to the reported rate in the tests. However, if I copy my lines slightly changing the cuts (a really small difference to avoid errors), append them to b_to_xll_spruce.py(here) and to b_to_xll_spruce_hhll.py, and I rerun the previous command I obtain:

"Monitor_SpruceRD_B03HpHmEE_SameSign_n_candidates": 43
"Monitor_SpruceRD_B03HpHmEE_n_candidates": 37
"Monitor_SpruceRD_B0ToHpHmEE_SameSign_n_candidates": 43
"Monitor_SpruceRD_B0ToHpHmEE_n_candidates": 37
"Monitor_SpruceRD_B03HpHmMuMu_n_candidates": 25
"Monitor_SpruceRD_B03HpHmMuMu_SameSign_n_candidates": 22
"Monitor_SpruceRD_B0ToHpHmMuMu_n_candidates": 25
"Monitor_SpruceRD_B0ToHpHmMuMu_SameSign_n_candidates": 22
"Monitor_SpruceRD_BuToHpEE_SameSign_n_candidates": 21
"Monitor_SpruceRD_Bu3HpEE_SameSign_n_candidates": 21
"Monitor_SpruceRD_Bu3HpEE_n_candidates": 15
"Monitor_SpruceRD_Bu3HpMuMu_n_candidates": 15
"Monitor_SpruceRD_BuToHpEE_n_candidates": 15
"Monitor_SpruceRD_BuToHpMuMu_n_candidates": 15
"Monitor_SpruceRD_Bu3HpMuMu_SameSign_n_candidates": 3
"Monitor_SpruceRD_BuToHpMuMu_SameSign_n_candidates": 3
"Monitor_SpruceRD_Bu2HpMuMu_n_candidates": 0
"Monitor_SpruceRD_Bu2HpEE_SameSign_n_candidates": 0
"Monitor_SpruceRD_Bu2HpEE_n_candidates": 0
"Monitor_SpruceRD_Bu2HpMuMu_SameSign_n_candidates": 0
"Monitor_SpruceRD_B02HpHmMuMu_n_candidates": 0
"Monitor_SpruceRD_B02HpHmMuMu_SameSign_n_candidates": 0
"Monitor_SpruceRD_B02HpHmEE_n_candidates": 0
"Monitor_SpruceRD_B02HpHmEE_SameSign_n_candidates": 0

This effect can be observed by uncommenting the lines here and here.

Observation N.2

Secondly, I have observed another issue. If I modify spruce_bandwidth_wg_streams_smalltest.py to use rather than make_module_streams (here), make_streams from here, into the script spruce_bandwidth_wg_streams_smalltest_var.py, the number of candidates now changes to:

"Monitor_SpruceRD_Bu3HpEE_n_candidates": 15
"Monitor_SpruceRD_B02HpHmEE_SameSign_n_candidates": 43
"Monitor_SpruceRD_B03HpHmEE_SameSign_n_candidates": 43
"Monitor_SpruceRD_B02HpHmEE_n_candidates": 37
"Monitor_SpruceRD_B0ToHpHmEE_SameSign_n_candidates": 43
"Monitor_SpruceRD_B0ToHpHmEE_n_candidates": 37
"Monitor_SpruceRD_B03HpHmEE_n_candidates": 37
"Monitor_SpruceRD_B02HpHmMuMu_SameSign_n_candidates": 22
"Monitor_SpruceRD_B02HpHmMuMu_n_candidates": 25
"Monitor_SpruceRD_Bu2HpEE_SameSign_n_candidates": 21
"Monitor_SpruceRD_B0ToHpHmMuMu_SameSign_n_candidates": 22
"Monitor_SpruceRD_B0ToHpHmMuMu_n_candidates": 25
"Monitor_SpruceRD_B03HpHmMuMu_n_candidates": 25
"Monitor_SpruceRD_B03HpHmMuMu_SameSign_n_candidates": 22
"Monitor_SpruceRD_Bu3HpEE_SameSign_n_candidates": 21
"Monitor_SpruceRD_BuToHpEE_SameSign_n_candidates": 21
"Monitor_SpruceRD_Bu3HpMuMu_n_candidates": 15
"Monitor_SpruceRD_Bu2HpEE_n_candidates": 15
"Monitor_SpruceRD_Bu2HpMuMu_n_candidates": 14
"Monitor_SpruceRD_BuToHpEE_n_candidates": 15
"Monitor_SpruceRD_BuToHpMuMu_n_candidates": 15
"Monitor_SpruceRD_Bu2HpMuMu_SameSign_n_candidates": 3
"Monitor_SpruceRD_Bu3HpMuMu_SameSign_n_candidates": 3
"Monitor_SpruceRD_BuToHpMuMu_SameSign_n_candidates": 3

when running:

run gaudirun.py Hlt/Hlt2Conf/tests/options/bandwidth/spruce_bandwidth_input.py Hlt/Hlt2Conf/tests/options/bandwidth/spruce_bandwidth_wg_streams_smalltest_var.py

I am a bit puzzled because I would have naively not expected any change between these different approaches. I can provide some of the logs/scripts if needed. As a comment you can find the corresponding logs.