Skip to content

[RTADPA BW Tests] Take account of compression changes at Hlt2/Sprucing

Luke Grazette requested to merge lugrazet-bw-compression into master

We use os.stat(fpath).st_size to extract the size of the streams in Bytes. (For Hlt2 we also run zstd -5 {fpath} to compress the Hlt2 output and extract that filesize too).

Then in line-and-stream-rates we calculate the compression factor and use this when calculating the avg evt size / dstdata size per line and per stream.

  • Hlt2: zstd compressed fsize / default fsize
  • Spruce: default fsize / sum(size(rawbanks)) for file.

cc @rjhunter @shunan

Locally evaluated changes

Evaluated ['these changes', 'these changes + forcing compression_factor=1'] with a reduced set of Hlt2 lines (to reduce load on local testing).

Can see:

hlt2_bandwidth

Compression factor set to 1:
uncompressed__hlt2__bandwidth_bar_chart__production__headline

Compression factor evaluated correctly: hlt2__bandwidth_bar_chart__production__headline

spruce_bandwidth

We recover reasonable looking output from the spruce test instead of the ~3x larger output seen on the UpgradeRateTest pages on master.

spruce__bandwidth_bar_chart__wg__headline

ToDo:

Edited by Luke Grazette

Merge request reports