Update BandQ inclusive line data processing flow, turbo PersistReco -> full stream + full dst sprucing

This MR is for minimising the impact of overlap between BandQ turbo lines and turbo from other WGs.

What we learned recently is that, if a event is triggered by B&Q persistReco turbo and another non-persistReco from other WG, the full event will be saved twice in both B&Q stream and another stream. It is not necessary to double-use our disk for the same event.

As a temporary solution, I would try to move some of the strongly-overlapping persistReco lines to Full stream, and use sprucing framework to make it accessible to users.

Before merging there are several checks to do:

1. Use MB MC to go through the hlt2 -> sprucing. Try to make sure that this MR does not introduce any bugs that will kill signals during the process.
2. Test the Impact on the bandwidth cost of other WGs and see if this change can really help to save the disk space.
3. Understand the reason of increasing of the SLB sprucing bandwidth cost. (c.c. to @anmorris )

dataflow test

Now we are using the same selection for Hlt2 full stream b-spectroscopy/DoubleCharm/BandQ_Dimuon lines and sprucing b-spectroscopy/DoubleCharm/BandQ_Dimuon lines. The output rate of the hlt2 lines and sprucing lines should be the same. This can be checked by comparing the output of hlt2__all_rates.html and spruce__all_rates.html produced by stack/DBASE/PRConfig/scripts/benchmark-scripts/Moore_hlt2_and_spruce_bandwidth.sh (or using their origin files: hlt2__rates_per_line__production__full.csv and spruce__rates_per_line__wg__bandq.csv). Following numbers are generated using this python script:read_rate.py

Double-Charm lines. Rate in kHz

BandQ_DoubleCharmOppositeSignDecision , hlt2 rate = 0.1824 , spruce rate = 0.1824 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharmSameSignDecision , hlt2 rate = 0.0912 , spruce rate = 0.0912 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToHHDecision , hlt2 rate = 0.0798 , spruce rate = 0.07979999999999998 , hlt2 rate - spruce rate = 1.3877787807814457e-17
BandQ_DoubleCharm_D0ToHHHHDecision , hlt2 rate = 0.0456 , spruce rate = 0.0456 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToHHHH_D0ToHHDecision , hlt2 rate = 0.0456 , spruce rate = 0.0456 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToHHHH_D0ToKsDDHHDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToHHHH_D0ToKsLLHHDecision , hlt2 rate = 0.10260000000000001 , spruce rate = 0.1026 , hlt2 rate - spruce rate = 1.3877787807814457e-17
BandQ_DoubleCharm_D0ToKsDDHHDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToKsDDHH_D0ToHHDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToKsLLHHDecision , hlt2 rate = 0.17099999999999999 , spruce rate = 0.17099999999999999 , hlt2 rate - spruce rate = 0.0
BandQ_DoubleCharm_D0ToKsLLHH_D0ToHHDecision , hlt2 rate = 0.21660000000000001 , spruce rate = 0.2166 , hlt2 rate - spruce rate = 2.7755575615628914e-17
BandQ_DoubleCharm_D0ToKsLLHH_D0ToKsDDHHDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0

b-spectroscopy lines. Rate in kHz

BandQ_BcForSpectroscopyDecision , hlt2 rate = 0.0228 , spruce rate = 0.0228 , hlt2 rate - spruce rate = 0.0
BandQ_BcForSpectroscopySLDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_BdForSpectroscopyDecision , hlt2 rate = 0.17099999999999999 , spruce rate = 0.17099999999999999 , hlt2 rate - spruce rate = 0.0
BandQ_BsForSpectroscopyDecision , hlt2 rate = 0.114 , spruce rate = 0.114 , hlt2 rate - spruce rate = 0.0
BandQ_BsForSpectroscopySLDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_BuForSpectroscopyDecision , hlt2 rate = 0.10260000000000001 , spruce rate = 0.1026 , hlt2 rate - spruce rate = 1.3877787807814457e-17
BandQ_BudForSpectroscopySLDecision , hlt2 rate = 0.3078 , spruce rate = 0.30779999999999996 , hlt2 rate - spruce rate = 5.551115123125783e-17
BandQ_LbForSpectroscopyDecision , hlt2 rate = 0.0114 , spruce rate = 0.0114 , hlt2 rate - spruce rate = 0.0
BandQ_LbForSpectroscopySLDecision , hlt2 rate = 0.0684 , spruce rate = 0.06839999999999999 , hlt2 rate - spruce rate = 1.3877787807814457e-17
BandQ_OmegabForSpectroscopyDecision , hlt2 rate = 0.0228 , spruce rate = 0.0228 , hlt2 rate - spruce rate = 0.0
BandQ_OmegabForSpectroscopySLDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_Xib0ForSpectroscopyDecision , hlt2 rate = 0.0342 , spruce rate = 0.034199999999999994 , hlt2 rate - spruce rate = 6.938893903907228e-18
BandQ_XibForSpectroscopySLDecision , hlt2 rate = 0.0114 , spruce rate = 0.0114 , hlt2 rate - spruce rate = 0.0
BandQ_XibmForSpectroscopyDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0

BandQ dimuon lines influenced by this MR. rate in kHz

BandQ_DiMuonIncDecision , hlt2 rate = 0.114 , spruce rate = 0.114 , hlt2 rate - spruce rate = 0.0
BandQ_DiMuonIncHighPTDecision , hlt2 rate = 0.28500000000000003 , spruce rate = 0.285 , hlt2 rate - spruce rate = 5.551115123125783e-17
BandQ_DiMuonSameSignHighMassDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_DiMuonSameSignIncDecision , hlt2 rate = 0.0798 , spruce rate = 0.07979999999999998 , hlt2 rate - spruce rate = 1.3877787807814457e-17
BandQ_DiMuonSameSignIncHighPTDecision , hlt2 rate = 0.0684 , spruce rate = 0.06839999999999999 , hlt2 rate - spruce rate = 1.3877787807814457e-17
BandQ_DiMuonSoftDecision , hlt2 rate = 0.0 , spruce rate = 0.0 , hlt2 rate - spruce rate = 0.0
BandQ_DiMuonUpsilonDecision , hlt2 rate = 0.0798 , spruce rate = 0.07979999999999998 , hlt2 rate - spruce rate = 1.3877787807814457e-17

bandwidth test

The impact of bandwidth tested using stack/DBASE/PRConfig/scripts/benchmark-scripts/Moore_hlt2_and_spruce_bandwidth.sh. In the following part, Without this MR means the master version based on which the MR is created; With this MR means the test using the settings in this MR.

Without this MR, Hlt2 bandwidth situation (taken from hlt2__line_descriptives.html):

Without this MR, Sprucing bandwidth situation (taken from spruce__rates_for_all_streams__wg.html):

With this MR, Hlt2 bandwidth situation:

With this MR, Sprucing bandwidth situation:

Summary of impact of tape bandwidth cost:

Turbo changes from 2.76GB/s to 2.58GB/s

Full changes from 7.33GB/s to 7.44GB/s

Total: Reduce (2.76+7.33) - (2.58+7.44) = 0.07 GB/s

Summary of impact of disk bandwidth cost:

Turbo changes from 4.2GB/s to 3.73GB/s

Sprucing changes from (0.845+0.256+0.14+0.119+0.0793+0.0162+0.000205) = 1.4557 GB/s to (0.853+0.359+0.257+0.142+0.0805+0.0164+0.000233) = 1.708GB/s .

Total: Reduce (4.21+1.4557) - (3.73+1.708) = 0.2277 GB/s

The issue of increasing of SLB sprucing bandwidth cost.

We had a discussion in mattermost https://mattermost.web.cern.ch/lhcb/pl/pibudtkfufgwxdakfheahcju5e about a 0.05GB/s increase of SLB sprucing lines, which was considered to be related to this MR.

Now after doing an additional local test, using the master branch version associated with this MR, we find the 0.05GB/s variation disappear. What triggered the discussion in the mattermost channel should be mainly due to a mismatching of the software version of the local test and nightly test.

Based on the updated local test, it seems that this MR will generate a 1% increase of event size of all sprucing lines in other WGs. To be checked whether it can be reproduced in the ci-test; if yes, whether this is an expected behavior.