Skip to content

Test counters and histograms via JSON sink and improve testing fluctuating counters

Change the way how histograms and counters are tested in Pytest to use JSON sink instead of parsing counter and histogram tables from the standard output. The JSON file contains more information about counters and histograms than the summary tables, except statistical metrics which are missing and are computed in numpy by Pytest. The structures of the counters and histograms blocks in YAML reference are adjusted to match those from JSON. For histograms the field axis (with bins and labels inside) is not added to the reference.

To setup the JSON sink, PyConf assigns a unique name f"{os.getenv('PYTEST_NAME')}_hist_counts.json" to monitoring_file option that configures the JSON sink, but only if PYTEST_NAME exists and the monitoring_file was not configured before.

Pytest's counters and histos fixtures call a new _detect_json_sink fixture that checks if Gaudi__Monitoring__JSONSink writes JSON file into the stdout. If not, they will find and process the tables from the stdout. This is needed for the backward compatibility if a test, by any reason, is not able to use the JSON sink.

Example, old version, 2D histogram:

    2D:
      errorsPerBankLocation:
      - errorsPerBankLocation
      - Errors per Bank Location
      - Ents/All=  600/600  <X>/sX=18.167/13.234,<Y>/sY=2.6667/2.1344

New version (values are rounded to 6th decimal place by pytest) :

   errorsPerBankLocation:
      dimension: 2
      nEntries: 600
      title: Errors per Bank Location
      sum: 600
      meanX: 18.166667
      stdDevX: 13.234005
      meanY: 2.666667
      stdDevY: 2.134375

Old version, counters:

  MDFIOAlg:
    set: MDFIOAlg
    total_counters: 1
    '#banks in raw event':
      binomial: false
      name: '#banks in raw event'
      count: 100
      sum: 62901.0
      mean/eff^*: 629.01
      rms/eff^*: 0.099499
      min: 629.0
      max: 630.0

New version:

  MDFIOAlg:
    '#banks in raw event':
      max: 630
      mean: 629.01
      min: 629
      nEntries: 100
      standard_deviation: 0.099499
      sum: 62901

Enhanced the way of declaring excluded and fluctuating counters to:

  1. Allow using regex pattern instead of full names
  2. Allow exclude counters only in specified algorithms
  3. Fluctuating counters can now fluctuate both in sum and nEntries
  4. Fix a bug with abs() missing when checking if fluctuation is inside the allowed margin
    Supported way of excluding the counters:
    exclude_counters = {
                        r".*rithm1": [r"Funny counter", r".*ounter to exclu.*"],
                        r".*rithm2": r"Dancing counter",
                        }
    Supported way of declaring the fluctuating counters:
    fluctuating_counters = {r".*Table2": {r"Flu.*": {"nEntries": 2, "sum": 0.5}}}
    what stands for:
        r".*Table2" - title pattern for the counter block,
        r"Flu.*" - name pattern for the counter,
        "nEntries" and "sum" fields contain allowed fluctuation off the reference

This MR can go together with Rec!4283

Edited by Pawel Kopciewicz

Merge request reports

Loading