Optimise basket sizes created by RootCnvSvc

Previously the basket size configuration could only be coarsely applied to all branches but in most real world files the size of the data in each branch varies significantly. Setting a small value causes large branches to be packed into individual baskets and prevents effective compression. Setting a large value improves the compression ratio of large branches but causes excessive memory usage.

Instead we change the parameters to only specifiy the maximum and minimum branch size and provide a target for the number of events to pack into each basket. We then use the first event to estimate the size of the branch.

This makes the branching very sensitive to the first event size however this seems to work okay in practice. If needed the basket sizes could be refined after several events to give a more precise estimate.

marked this merge request as ready

changed milestone to %v36r9

added lhcb-gaudi-head label

assigned to @clemenci

• [2022-11-08 14:22] Validation started with lhcb-gaudi-head#3388
• [2022-11-09 00:02] Validation started with lhcb-gaudi-head#3389
• [2022-11-09 00:21] Validation started with lhcb-run2-gaudi-head#413
• [2022-11-09 15:43] Validation started with lhcb-gaudi-head#3390
• [2022-11-09 16:10] Validation started with lhcb-gaudi-head#3391
• [2022-11-09 18:02] Validation started with lhcb-gaudi-head#3392
• [2022-11-15 13:59] Validation started with lhcb-gaudi-head#3399
• [2022-11-16 00:02] Validation started with lhcb-gaudi-head#3400
• [2022-11-16 00:21] Validation started with lhcb-run2-gaudi-head#418
• [2022-11-17 00:03] Validation started with lhcb-gaudi-head#3401
• [2022-11-17 00:21] Validation started with lhcb-run2-gaudi-head#419

removed lhcb-gaudi-head label

added 2 commits

• 5feebc08 - Optimise basket sizes created by RootCnvSvc
• eef8866f - Make GaudiExamples.return_codes.corrupted_input more reliable

Compare with previous version

added lhcb-gaudi-head label

I'm removing the label because I want to see if this is the reason for the failure https://lhcb-nightlies.web.cern.ch/nightly/lhcb-gaudi-head/3392/LHCb/x86_64_v2-centos7-gcc11-opt/tests#FSRAlgs.mergesmallfiles

VMem usage too high, 25.612842559814453 GB. Note that anything over 1.7 GB represents a serious problem.

removed lhcb-gaudi-head label

added lhcb-gaudi-head label

added 55 commits

• eef8866f...5de4967d - 53 commits from branch gaudi:master
• 59218fc9 - Optimise basket sizes created by RootCnvSvc
• 379db197 - Make GaudiExamples.return_codes.corrupted_input more reliable

Compare with previous version

mentioned in commit lhcb/LHCb@b03cc1e1

mentioned in merge request lhcb/LHCb!3871 (merged)

resolved all threads

approved this merge request

merged

mentioned in commit 9d873402