Skip to content

[RTA/DPA BW Tests] Correct compression level for HLT2 and Sprucing in BW Tests

Luke Grazette requested to merge lugrazet-BWTest-compressionsettings into master

Short descr:

  • zstd -5 -> zstd -3 for HLT2 compression estimation.
  • explicitly require LZMA:4 for Sprucing compression level.

HLT2 Changes

Looking at Rosen's script (/home/rmatev/piquet/stream.py in the Online network), --compress simply does zstd -q -f -o filename, i.e. doesn't specify a compression level.

This would be equivalent to zstd -3, c.f. zstd -5 which we use currently in the BW Tests.
This should make negligible difference, but we want to be as accurate to the data conditions as we can be.

Difference

over some test number of events, they have similar compression factors. 33.8% and 34.8%. Thus we expect this will increase the result of the BWTests by a small margin, but if it's more accurate to data-taking then we want to include it.

[phrhbm@veleta modern_stack]$ zstd -5 tmp/MDF/hlt2_bw_testing__production__full.mdf 
tmp/MDF/hlt2_bw_testing__production__full.mdf : 33.82%   (  12.5 MiB =>   4.22 MiB, tmp/MDF/hlt2_bw_testing__production__full.mdf.zst) 

[phrhbm@veleta modern_stack]$ zstd tmp/MDF/hlt2_bw_testing__production__full.mdf 
tmp/MDF/hlt2_bw_testing__production__full.mdf : 34.76%   (  12.5 MiB =>   4.34 MiB, tmp/MDF/hlt2_bw_testing__production__full.mdf.zst)

Sprucing Changes

Addressing #19 (closed), requires lhcb/LHCb!4661 (closed) by adding compression of:

from Configurables import RootCnvSvc
RootCnvSvc().GlobalCompression = "LZMA:4"

Actually adds the usage via an extra option ${APPCONFIGOPTS}/Persistency/Compression-LZMA-4.py to prevent duplication.

Difference

I see some percent level movements in the sizes from local testing, but very insignificant. Should be verified again in ci-test on master.

ToDo:

  • Check locally
  • Check that Hlt2 is now following what the data movers do? (Chris Burr / Frederic Hemmer)
  • Confirm with Sprucing Team the compression settings being used : #19 (comment 8389289)
  • Sign off by DPA that the sprucing change is understood
    • Opened #839 to investigate further
  • Check in ci-test with Moore lhcb/Moore!3478 now that master is ready : !446 (comment 8740230)
Edited by Luke Grazette

Merge request reports