RecJobTransforms + SimuJobTransforms: Switch the compression of all temporary files to ZLIB to speed up reading/writing

As we're discussing in ATEAM-656, this MR changes the compression algorithm for all temporary files to ZLIB. In this context, a file is temporary if either of the following two criteria is met:

• The transform runs a chained workflow but has no --outputXYZFile specified for the intermediate step(s), e.g. RAWtoESD followed by ESDtoAOD without specifying --outputESDFile (this is not being done)
• The files that are written out by the worker processes in AthenaMP (this is already being done)

In the first case, the output filename is set to be tmp.XYZ where XYZ stands for the appropriate step, while in the second case _000 is appended to the file name, both by convention.

From a quick test based on q431 w/ 50 events, here is the comparison of StreamESD performance, as well as resulting ESD file sizes, by different compression schemes (compression level is always set to 1):

Compression	File Size [MB]	CPU-time [sec/evt]	Note
LZMA	139 MB	855	Leading CPU consumer
ZLIB	180 MB	371	4th leading CPU consumer
ZSTD	181 MB	287	4th leading CPU consumer
LZ4	245 MB	221	4th leading CPU consumer

Again, we're not proposing to change the compression scheme for permanent files (which is LZMA for all upstream formats including AODs and - at least for the time being - ZLIB for DAODs), only for the temporary ones. Going from ZLIB to LZ4 would increase the file size by about 35% while improving the StreamESD CPU performance by 40%. In all three cases, ZLIB, ZSTD, and LZ4, the ESDtoAOD performances are practically the same in this test.

This should especially help w/ high thread count AthenaMT jobs w/ chained workflows where the temporary intermediate files are currently being compressed w/ LZMA, which is very expensive.

Thanks @amete

These are good enough numbers for changing the compression of temporary files. I think that ZLIB performance is adequate (LZ4 has too large of a size increase that may give trouble with overall file size). [I don't object to ZSTD, which is a little faster, either, but default to familiarity of ZLIB]

So I would recommend using ZLIB on "_000" and "tmp." files in both the existing RecJobTransforms and new SimuJobTransforms customizations.

I suggest limiting this MR to those changes (so it can be done quickly) and do further optimization (especially on RDO) in a future MR.

Peter

added 1 commit

• 763cfb5d - Revert temporary file compression from LZ4 to ZLIB

Compare with previous version

changed title from WIP: RecJobTransforms + SimuJobTransforms: Switch the compression of temporary files from ZLIB to LZ4 to speed up reading/writing to WIP: RecJobTransforms + SimuJobTransforms: Switch the compression of all temporary files to ZLIB to speed up reading/writing

changed the description

added 1 commit

• b5477eda - Revert temporary file compression from LZ4 to ZLIB

Compare with previous version

Thanks a lot @gemmeren. I agree, let's go w/ ZLIB. I just made the necessary updates. Please let m know if you have further comments. I'm removing the WIP now.

unmarked as a Work In Progress

This merge request affects 2 packages:

• Reconstruction/RecJobTransforms
• Simulation/SimuJobTransforms

Adding @goetz ,@jchapman ,@vpascuzz as watchers

added Reconstruction Simulation master review-pending-level-1 labels

changed the description

CI Result SUCCESS (hash b5477eda)

	Athena	AthSimulation	AthGeneration	AnalysisBase
externals
cmake
make
required tests
optional tests

Full details available on this CI monitor view
Athena: number of compilation errors 0, warnings 0
AthSimulation: number of compilation errors 0, warnings 0
AthGeneration: number of compilation errors 0, warnings 0
AnalysisBase: number of compilation errors 0, warnings 0
For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 21298]

added review-approved label and removed review-pending-level-1 label

merged

mentioned in commit f536859c

added sweep:ignore label