Test counters and histograms via JSON sink and improve testing fluctuating counters
Change the way how histograms and counters are tested in Pytest to use JSON sink instead of parsing counter and histogram tables from the standard output. The JSON file contains more information about counters and histograms than the summary tables, except statistical metrics which are missing and are computed in numpy by Pytest. The structures of the counters and histograms blocks in YAML reference are adjusted to match those from JSON. For histograms the field axis (with bins and labels inside) is not added to the reference.
To setup the JSON sink, PyConf assigns a unique name f"{os.getenv('PYTEST_NAME')}_hist_counts.json" to monitoring_file option that configures the JSON sink, but only if PYTEST_NAME exists and the monitoring_file was not configured before.
Pytest's counters and histos fixtures call a new _detect_json_sink fixture that checks if Gaudi__Monitoring__JSONSink writes JSON file into the stdout.
If not, they will find and process the tables from the stdout. This is needed for the backward compatibility if a test, by any reason, is not able to use the JSON sink.
Example, old version, 2D histogram:
2D:
errorsPerBankLocation:
- errorsPerBankLocation
- Errors per Bank Location
- Ents/All= 600/600 <X>/sX=18.167/13.234,<Y>/sY=2.6667/2.1344
New version (values are rounded to 6th decimal place by pytest) :
errorsPerBankLocation:
dimension: 2
nEntries: 600
title: Errors per Bank Location
sum: 600
meanX: 18.166667
stdDevX: 13.234005
meanY: 2.666667
stdDevY: 2.134375
Old version, counters:
MDFIOAlg:
set: MDFIOAlg
total_counters: 1
'#banks in raw event':
binomial: false
name: '#banks in raw event'
count: 100
sum: 62901.0
mean/eff^*: 629.01
rms/eff^*: 0.099499
min: 629.0
max: 630.0
New version:
MDFIOAlg:
'#banks in raw event':
max: 630
mean: 629.01
min: 629
nEntries: 100
standard_deviation: 0.099499
sum: 62901
Enhanced the way of declaring excluded and fluctuating counters to:
- Allow using regex pattern instead of full names
- Allow exclude counters only in specified algorithms
- Fluctuating counters can now fluctuate both in sum and nEntries
- Fix a bug with
abs()missing when checking if fluctuation is inside the allowed margin
Supported way of excluding the counters:
exclude_counters = {
r".*rithm1": [r"Funny counter", r".*ounter to exclu.*"],
r".*rithm2": r"Dancing counter",
}
Supported way of declaring the fluctuating counters:
fluctuating_counters = {r".*Table2": {r"Flu.*": {"nEntries": 2, "sum": 0.5}}}
what stands for:
r".*Table2" - title pattern for the counter block,
r"Flu.*" - name pattern for the counter,
"nEntries" and "sum" fields contain allowed fluctuation off the reference
This MR can go together with Rec!4283