Create 2D histograms for ToT, RelBCID, and Occupancy (one 1D hist per scan parameter)

• ToT histogram include parameter dimension (add 2D histogram plot)
• relBCID histogram include parameter dimension (add 2D histogram plot)
• occupancy data structure include parameter dimension (no scurve histogram needed)

Breaking change to data format needs recreation of almost all fixtures.

Consistant data stuctures for histograms.

changed the description

added 1 commit

• 5d23864d - MAINT:fixing lint

Compare with previous version

changed title from Add 2D ToT histogram for threshold scan (injection step) to Create 2D ToT histogram (one 1D hist per scan parameter)

Yes, this can be done and simplifies all analysis (e.g. tot calibration does not have to create this on it own). If we go down the road of creating histograms per scan parameter, we should not stop here but also do this for the rel_bcid/occupancy histogram. Scurve hist will become obsolete if occupancy histogramming is done per scan parameter id. I think that is a good idea.

approved this merge request

added 1 commit

• 18afd5b2 - ENH: add relBCID param scan historgram

Compare with previous version

changed title from Create 2D ToT histogram (one 1D hist per scan parameter) to Create 2D histograms for ToT and RelBCID (one 1D hist per scan parameter)

changed the description

@dphol Let me know if something like this is ok. If yes please help with fixtures (my OS seems to old ;-( ).

added 1 commit

• d2a3e98a - MAINT: linting

Compare with previous version

added 1 commit

• 3b28ab3a - CI: fix tests

Compare with previous version

added 1 commit

• 6311a663 - MAINT: lint

Compare with previous version

added 1 commit

• 3ab66f8d - MAINT: lint

Compare with previous version

added Enhancement label

changed title from Create 2D histograms for ToT and RelBCID (one 1D hist per scan parameter) to Create 2D histograms for ToT, RelBCID, and Occupancy (one 1D hist per scan parameter)

added 1 commit

• 172b8c55 - BUG: fix test

Compare with previous version

mentioned in merge request !171 (closed)

It looks good to me. One could think about using a dtype (uint8-uint32) for the 2D hists that depends on the number of parameters to be more sure that the hist fits into RAM.

The BCID histogram e.g. can be quite big:

100 * 400 * 192*4 byte * 32 ~ 1Gb

With the rule of thumb to be able to have it 2x in RAM for data analysis that is quite a lot. With dtype=uint8 that is sufficient here we can quarter the memory foot print.

Since array creation is done at one position in the code, this feature is easy to implement:

# Prevent arrays > 250 Mb for up to 100 different parameter settings to ease in RAM analysis
if n_scan_params < 25:
  dtype_large_hist = np.uint32
elif n_scan_params < 50:
  dtype_large_hist = np.uint16
else:
  dtype_large_hist = np.uint8
hist_occ = np.zeros(shape=(400, 192, n_scan_params), dtype=np.uint32)
hist_tot = np.zeros(shape=(400, 192, n_scan_params, 16), dtype=dtype_large_hist)
hist_rel_bcid = np.zeros(shape=(400, 192, n_scan_params, 32), dtype=dtype_large_hist)

Log some warning?