Skip to content

RecoMon savesets randomly replacing `-` with `.`

Since a long time we have issues with two very particular histograms in Reco{X}Mon: the RICH/RiCKResLong/Rich2Gas/ASide-left and RICH/RiCKResLong/Rich2Gas/CSide-right. From time to time (randomly every ~200-300 runs maybe) the ByRun histograms are empty. Finally I took a bit of a closer look and found that these are empty because the savesets by time stamp are not coherent. Let me give run 304682 as example. There are two original savesets produced and they have different histogram names in the directories

bash-5.2$ root -l /hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223332.root
root [0] 
Attaching file /hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223332.root as _file0...
(TFile *) 0x5567336bca30
root [1] _file0->cd("RICH/RiCKResLong/Rich2Gas/ASide-left")
Error in <TDirectoryFile::cd>: Unknown directory ASide-left
(bool) false
root [2] _file0->ls()
TFile**		/hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223332.root	
 TFile*		/hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223332.root	
  TDirectoryFile*		RICH	RICH
   TDirectoryFile*		RiCKResLong	RiCKResLong
    TDirectoryFile*		Rich2Gas	Rich2Gas
     KEY: TDirectoryFile	ASide.left;1	ASide.left
     KEY: TDirectoryFile	CSide.right;1	CSide.right

and

bash-5.2$ root -l /hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223646-EOR.root
root [0] 
Attaching file /hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223646-EOR.root as _file0...
(TFile *) 0x55b24124b0e0
root [1] _file0->cd("RICH/RiCKResLong/Rich2Gas/ASide-left")
(bool) true
root [2] _file0->ls()
TFile**		/hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223646-EOR.root	
 TFile*		/hist/Savesets/2024/LHCb/RecoPhysMon/08/26/RecoPhysMon-304682-20240826T223646-EOR.root	
  TDirectoryFile*		RICH	RICH
   TDirectoryFile*		RiCKResLong	RiCKResLong
    TDirectoryFile*		Rich2Gas	Rich2Gas
     TDirectoryFile*		ASide-left	ASide-left
      KEY: TH1D	ckResAllPerPanel;1	Rich2Gas ASide-left Rec-Exp CKTheta - All photons
      KEY: TH1D	ckResAllPerPanelBetaCut;1	Rich2Gas ASide-left Rec-Exp CKTheta - All photons - 0.999900<beta
      KEY: TH1D	ckResAllPerPanelHTypePD;1	Rich2Gas ASide-left Rec-Exp CKTheta - All photons - H-Type PDs
      KEY: TH1D	ckResAllPerPanelRTypePD;1	Rich2Gas ASide-left Rec-Exp CKTheta - All photons - R-Type PDs
      KEY: TDirectoryFile	cols;1	cols
     KEY: TDirectoryFile	ASide-left;1	ASide-left
     KEY: TDirectoryFile	CSide-right;1	CSide-right

Apparently at the time of the Saveset creation there is something weird happening that replaces randomly the - in the saveset name by .

I have no idea so far as to why this may happen, but a few random pointers that may help closing in on the issue:

  • The problem was so far always and solely spotted on these two histograms
  • It may (or may not) have to do with the - in the name
  • The RICH monitoring algorithms still use the old and non-thread safe AIDA histograms
  • The issue may (or may not) be related to Online#5 which observed a different problem with the same histograms

Maybe @frankm @rmatev have an idea. Also tagging @fbetti and @jonrob who have brought up the topic several times For reference, the implementation of the monitoring algorithm is https://gitlab.cern.ch/lhcb/Rec/-/blob/master/Rich/RichFutureRecMonitors/src/RichSIMDPhotonCherenkovAngles.cpp

Online!1068 (merged) may fix this

Edited by Titus Mombaecher