adding modifications for re-reco
General cleanup:
- config files
- snakemake configs
- updating the step_*cfg.py files
Improvements to NANO:
- throughput : fill tables with single loop plus two possibilities
- use reserve and push_back instead of resize
- declare columns as class variables and resize for first event + use
edm::Span
- energy and time as float (i really doubt we need a double)
- change compression algorithm
Some numerology on the NANO using pedestal run 1726225194 (measured in vocms1002 using maxThreads=4 and 50k events)
With the default LZMA compression 9 (as in the TB) I get the following. The major improvement I'm convinced is because instead of doing two loops (one over DIGIs and one over RecHits) one moves to a single loop.
Version | Throughput [ev/s] |
---|---|
Updated from !46 (merged) | 113 |
Move to reserve+push_back + 1 loop | 230 |
Move to 1/run+edm::Span + 1 loop |
230 |
However moving from LZMA compression to ZSTD the gain in throughput is remarkable. There's an increase in the file size (50%) but I would say it's still ok for our purposes.
Compression | Level | Throughput [ev/s] | Size [MB] (50k events) |
---|---|---|---|
LZMA | 9 | 230 | 118 |
LZMA | 5 | 248 | 118 |
ZSTD | 9 | 351 | 155 |
ZSTD | 5 | 1764 | 175 |
So in this merge request I propose to use for NANO:
-
edm::Span
for the filling (tbc if it pays off with ZS events) - ZSTD + compression 5
Maybe @hqu has additional suggestions.
Edited by Pedro Vieira De Castro Ferreira Da Silva