Change HltPackedDataWriter compression method

Currently, HltDataPackedWriter uses the LZMA-6 algorithm to compress data. However, the ZSTD-5 algorithm is more efficient for this application.

Local testing showed that, compared to LZMA-6, ZSTD-5 reduces the execution time of HltDataPackedWriter by a factor of 5 while increasing the compressed buffer size by only 6%.

This is similar to what is shown in slide 10 of this presentation.

FYI @sstahl @rmatev

marked this merge request as draft

changed the description

changed the description

you forgot one small thing:

diff --git a/Hlt/HltDAQ/src/component/HltPackedData.h b/Hlt/HltDAQ/src/component/HltPackedData.h
index b5b32785a..6916f26f2 100644
--- a/Hlt/HltDAQ/src/component/HltPackedData.h
+++ b/Hlt/HltDAQ/src/component/HltPackedData.h
@@ -64,7 +64,7 @@ namespace LHCb::Hlt::PackedData {
       sv.remove_prefix( 1 );
       sv.remove_suffix( 1 );
     };
-    constexpr auto algs = std::array{Compression::NoCompression, Compression::ZLIB, Compression::LZMA};
+    constexpr auto algs = std::array{Compression::NoCompression, Compression::ZLIB, Compression::LZMA, Compression::LZ4, Compression::ZSTD};
     auto           i    = std::find_if( algs.begin(), algs.end(), [sv]( auto a ) { return sv == toString( a ); } );
     if ( i == algs.end() ) return StatusCode::FAILURE;
     c = *i;

i.e. parse should be able to actually parse the values you added ;-)

added 1 commit

• 5cee89ff - Change HltPackedDataWriter compression method.

Compare with previous version

added 1 commit

• b1c9ca98 - Change HltPackedDataWriter compression method.

Compare with previous version

resolved all threads

resolved all threads

marked this merge request as ready

changed the description

approved this merge request

assigned to @rangel

@rangel This is ready for testing.

/ci-test --merge

added ci-test-triggered label

• [2021-08-17 15:40] Validation started with lhcb-master-mr#2719
• [2021-08-18 15:31] Validation started with lhcb-master-mr#2728

assigned to @lmeyerga and unassigned @rangel

added 1 commit

• 3ee85d3a - Change HltPackedDataWriter compression method.

Compare with previous version

resolved all threads

assigned to @rangel and unassigned @lmeyerga

/ci-test --merge --platforms=x86_64_v3-centos7-gcc10-opt

resolved all threads

assigned to @chasse

• [2021-08-19 10:04] Validation started with lhcb-master-mr#2731

approved this merge request

resolved all threads

merged

mentioned in commit 084b0bc6

added Persistency enhancement labels

mentioned in issue Moore#349