Skip to content
Snippets Groups Projects

Optimise basket sizes created by RootCnvSvc

Merged Chris Burr requested to merge lhcb/Gaudi:cburr/better-basket-sizing into master
  1. Nov 15, 2022
    • Chris Burr's avatar
    • Chris Burr's avatar
      Optimise basket sizes created by RootCnvSvc · 59218fc9
      Chris Burr authored
      Previously the basket size configuration could only be coarsely applied to all
      branches but in most real world files the size of the data in each branch varies
      significantly. Setting a small value causes large branches to be packed into
      individual baskets and prevents effective compression. Setting a large value
      improves the compression ratio of large branches but causes excessive memory
      usage.
      
      Instead we change the parameters to only specifiy the maximum and minimum
      branch size and provide a target for the number of events to pack into each
      basket. We then use the first event to estimate the size of the branch.
      
      This makes the branching very sensitive to the first event size however this
      seems to work okay in practice. If needed the basket sizes could be refined
      after several events to give a more precise estimate.
      59218fc9
Loading