Allen line monitoring counters different from decisions
When running Allen offline (using allen.py
), a discrepancy between the number of fired lines as reported by the RateValidator
(from DecReports) and the line_passes
counters (in GatherSelections) is observed for the GPU build (for non-prescaled lines). In particular, the counter reports lower values than the RateValidator
, while also the line_passes
counter is sometimes lower than the line_rates
- which doesn't make sense, since line_rates = line_passes * post_scale
. The corresponding monitoring histograms have the same values as the counters (again, lower than the DecReport values). The counter values are also not the same when running the same command more than once.
Using a CPU build, the two numbers agree and are stable when running more than one time.
I attach here the log files of the cpu and gpu builds, where both counter values and the rate validator numbers can be seen.
Fyi, I performed the tests with run 273190
(mdfs under /hlt2/objects/LHCb/0000273190/
) but also with a COLLISION23
MEP, and the issue is observed in both cases.
EDIT: After investigations from @dcampora the issue has been found and fixed here: !1294 (88d0b4c0)
A similar strategy should be adopted in all monitoring algorithms, to make sure they're deterministic. List of algorithms whose histograms I have found need adapting:
- ConsolidateSciFi
- ConsolidateSciFiSeeding
- ConsolidateMatchedTracks
- VeloConsolidateTracks
- MakeLongTrackParticles
- pv_beamline_cleanup
- D2KPiLine
- D2KKLine
- D2PiPiLine
- KsToPiPiLine
- DiMuonDrellYanLine
- DiMuonMassLine
- LowMassNoipDielectronLine
- DiMuonNoIPLine
- DisplacedDiMuonLine
- VeloKalmanFilter
- VertexFitter
- CaloFilterClusters