Skip to content

DerivationFrameworkPhys: Add postIncludes for testing Lossy Float Compression in DAOD_PHYS/DAOD_PHYSVAL production

This MR adds four postInclude files that'll ease testing lossy float compression during DAOD_PHYS/DAOD_PHYSVAL production. Each postInclude file is self-sufficient, defining a list of containers for which the lossy float compression should be applied, and configures the underlying stream appropriately. The usage is simply:

--postInclude 'AODtoDAOD:DerivationFrameworkPhys/postInclude.PHYS_LowLossyFloatCompression.py'

The compression lists are preliminary for testing (and identical between the files). They are simply built by looking at the top disk users in the DAOD_PHYS files in the most recently master nightlies in local tests. Needless to say, these can/will be refined as we go along.

For each derivation type, these files can be used in conjunction (i.e. included simultaneously), although at the moment this won't do anything more than simply adding the high compression one alone (because the lists are identical and the high compression has precedence).

In local tests, the low compression seems to reduce the DAOD_PHYS size by about 10% and the high compression by about 20% (500 events over a data file).

The eventual plan is to add the relevant configuration bits to DerivationFrameworkCore/SlimmingHelper.py so that the postInclude hooks will not be needed in production.

cc: @jcatmore @gemmeren @elmsheus

Edited by Alaettin Serhan Mete

Merge request reports