Skip to content

Fix file handling in CollectMaPSAs_DB.py

Andreas Matthias Nurnbergrequested to merge
nurnberg-cms/software/mapsa_analysis:desy_fix into production

This is intended to workaround a (severe) bug/issue in the way data files are retrieved in the analysis code. The issue we encountered at DESY is the following:

  1. The code tries to find all filenames belonging to a certain chip by calling (eg for the mask test for chip 10) ls ../Results_MPATesting/HPK_36234_042R/mpa_test_*_10_*_mask_test.csv here. Then (in get_recent() here in concatenates all digits it finds in the result file and returns the file with the maximum number. So far so good.
  2. In our case however this particular call results in the following two files being found:
  • ../Results_MPATesting/HPK_36234_042R/mpa_test_HPK_36234_042R_10_2025_05_21_13_05_44_mask_test.csv
  • ../Results_MPATesting/HPK_36234_042R/mpa_test_HPK_36234_042R_11_2025_05_21_13_10_02_mask_test.csv

Notice the fact that the second result file is actually coming from chip 11, and that it gets picked by coincidence because it was tested at 10 minutes past the hour. The way the most recent file gets identified (concat all digits and take maximum) results in the wrong file being picked for the analysis of chip 10, since the chip number is preceeding the timestamp. In that case, the highest chip number wins, irrespective of the actual timestamp.

As a workaround we changed the ls pattern to include a combination of mapsa_name and chip number, so that really only the timestamp is in the wildcard. This ensures that only the files belonging to the correct chip are selected.

With this finding, I cannot guarantee that the results and grades we uploaded so far are actually perfectly reflecting the state of the MaPSAs. We noticed the issue only because chip 11 of the above MaPSA got retested due to bad pixels on May 30 and the grading of the MaPSA still remained C although we were sure that all recent tests would result in grade A. For chip 10 the bad, outdated test of chip 11 was used in the grading.

This is most likely not a complete fix for the issue, since the get_recent(cmd) call is used vastly over many areas of the code. In some placed it already includes the combination of mapsa name and chip number, but I did not have the time to check all occurences.

(cc) @merkelp @drberry

Merge request reports