Skip to content

Add progress printout

Pieter David requested to merge piedavid/bamboo:printprogress into master

Closes https://gitlab.cern.ch/cp3-cms/bamboo/-/issues/49

Continuing the discussion from the issue: with IMT OnPartialResult will be triggered from the slowest thread, with its per-thread partial result. I wanted to prevent more synchronisation, so I'm just dividing the frequency by nThreads, multiplying the printed numbers with it (should be a reasonable estimate, slightly biased towards lower counts and rate values), and changed the default to every 12000 events (which can also be divided by 3). Based on the documentation, the callbacks need not be thread-safe. I also added a printout line from the start of the counter, which is the processing of the first event, because it can be useful for debugging.

Typical printout then:

INFO:bamboo.analysismodules:Sequential mode: calling processTrees for examples/postprocessing.py:PostprocessJME with (['/storage/data/cms/store/user/piedavid/topNanoAOD/v6-1-1/2017/TTTo2L2Nu_TuneCP5_PSweights_13TeV-powheg-pythia8/TopNanoAODv6-1-1_2017/200610_155044/0000/tree_206.root'], test_out/postproc_1/results/TTToSemilepton_2017.root, tree=Events, sample=TTToSemilepton_2017
INFO:bamboo.analysismodules:Enabling implicit MT for 3 threads
RDF event loop started to process entries
Processed 12000 entries. Elapsed: 29.7359s, average rate: 0.403553 kHz
Processed 24000 entries. Elapsed: 67.5434s, average rate: 0.355327 kHz
Processed 36000 entries. Elapsed: 89.0905s, average rate: 0.404083 kHz

(warning: new header, so also editable installations will need a pip install -e --upgrade)

Edited by Pieter David

Merge request reports