Update of muon showers line after retraining the BDT with HLT1_filtered data

Retraining of the BDT for the muon detector showers line Hlt2QEE_MDS_BDT_nHits with HLT1 filtered data to solve MC/data differences.

We tried to maintain reasonable bandwidth levels without the current prescale of the Hlt2QEE_MDS_BDT_nHits line (0.01) with two distinct processing lines:

• Full-stream – for background studies (a higher cut on the BDT was imposed).
• Turbo – to ensure signal retention if the full-stream cut is too high.

This merge request is associated with the approval of Update of muon-showers weights, which updates the BDT weights after retraining with HLT1-filtered data.

BANDWIDTH TESTS:

We conducted the QEE bandwidth test using 10.000 HLT1-filtered events for the following branches:

• qee_upgrade (old muon_showers line with a 0.01 prescale)

    -  Turbo bandwidth: 191.952 MB
    -  Full-stream bandwidth: 516.516 MB
    -  Rate: 516.0 kHz

• muon_showers (new Turbo and full-stream lines with no prescale):

    - Turbo bandwidth: 158.412 MB
    - Full-stream bandwidth: 523.74 MB
    - Turbo rate: 258.0 kHz
    - Full-stream rate: 129.0 kHz

added RTA qee labels

assigned to @cvazquez, @jcidvida, @mariape, and @amartin6

added 1 commit

• a47caa78 - pre-commit fixes

Compare with previous version

mentioned in merge request lhcb-datapkg/ParamFiles!135 (merged)

requested review from @isanders, @ahabdelm, and @paandreo

@rjhunter this supersedes !4582 (closed). Requested review to the DPA/RTA QEE liaisons.

added 1 commit

• 7a3207c5 - Corrected import locations and some redundancies in configs

Compare with previous version

added 1 commit

• ab6adc5e - pre-commit fixes

Compare with previous version

The full line needs a sprucing line in order for output to be written to disk, do you plan to add that now, or for EoY sprucing?

I think the BW estimates are a bit off compared to what I get (I am getting about 575 MB/s for qee_upgrade branch), are you remembering to compress the mdf file first?

mentioned in merge request !4367 (merged)

@amartin6 @mariape we've had a few MRs go in to qee_upgrade so this will need a rebase and might have some minor conflicts.

added 46 commits

• ab6adc5e...f7b83213 - 42 commits from branch qee_upgrade
• 4f8a1040 - Update of muon showers line after retraining the BDT with HLT1_filtered data
• 1ef45562 - pre-commit fixes
• ad929c4c - Corrected import locations and some redundancies in configs
• 2054f328 - pre-commit fixes

Compare with previous version

changed the description

Thanks very much for addressing comments so far.

Code looks good to me, just one comment on it for curiosity: why did you remove the monitoring from the turbo line and just have it on the full-stream line?

Bandwidth changes are fine.

added 1 commit

• ca913587 - Added monitoring for the full line

Compare with previous version

@mariape you seem to have, in your last commit, overwritten loads of the changes that @paandreo requested - I guess this was unintentional?

added 1 commit

• 5f060280 - Redid small changes in import locations and configs

Compare with previous version

added 1 commit

• ac00c766 - pre-commit fixes

Compare with previous version

mentioned in issue #939

resolved all threads

approved this merge request

@amartin6 @mariape @cvazquez @jcidvida since the QEE deadline has passed I want to move on here. The code looks good, and the bandwidth change you report here is fine.

Please please make a study of the robustness of this with data-taking conditions and add it to #939 by the end of next Tuesday so we have some time to react before qee_upgrade gets merged to master.

This can be merged now, I'll allow one of you to push the "merge" button.

resolved all threads

mentioned in commit e50ab1e8

merged