Threadsafe monitoring and checkers histograms

Most of the monitoring and checkers histograms are currently not using the thread-safe GaudiHisto's and need to be converted. See for example for TrackMonitor and TrackVertexMonitor: !2565 (merged).

For more reference material, see the slides from Sébastien: https://indico.cern.ch/event/1043362/contributions/4426284/attachments/2271504/3857794/MonitoringAndHistograms.pdf

To keep track of the algorithm that need to be converted, here's a list all algorithms currently running when do_data_monitoring=True or do_mc_checking=True.

Track monitors:

• TrackVertexMonitor
• TrackFitMatchMonitor
• TrackMonitor
• TrackPV2HalfAlignMonitor
• TrackV0Monitor
• TrackDiMuonMonitor

Calo monitors:

• CaloFutureClusterMonitor
• CaloFutureDigitMonitor
• CaloFutureTimeAlignment

Raw bank monitors:

• RawBankSizeMonitor

Track checkers:

• PrTrackChecker
• TrackResChecker

Hit(s) checkers:

• PrVPHitsChecker
• PrUTHitsChecker
• PrUTHitChecker (yes, this sounds identical to the one above but they're different?)
• PrFTHitsChecker

Multiplicity checkers:

• PrMultiplicityChecker

PV checker:

• PrimaryVertexChecker

IP resolution checker (not configured):

• TrackIPResolutionCheckerNT

RICH monitors:

• Rich__Future__Rec__Moni__DetectorHits
• Rich__Future__Rec__Moni__PixelClusters
• Rich__Future__Rec__Moni__DLLs
• Rich__Future__Rec__Moni__PhotonYield
• Rich__Future__Rec__Moni__GeometricalEfficiencies
• Rich__Future__Rec__Moni__MassHypoRings
• Rich__Future__Rec__Moni__SIMDPhotonCherenkovAngles
• Rich__Future__Rec__Moni__PixelBackgrounds
• Rich__Future__Rec__Moni__TrackRadiatorMaterial
• Rich__Future__Rec__Moni__TrackSelEff

@decianm @graven

added MC checking Monitoring labels

@valukash It seems that you are the author of PrUT/VP/FTHitsChecker. Would you have some time for this?

@decianm, unfortunately, I don't really have time until the 20ies of January or so. If it can wait I can do it at the end of January.

If this can be done until the end of January, it is fine.

Thanks!

@valukash

You think you will have time to work on this in the coming days?

Thanks!

Hi, @decianm! Yes, I am on to it.

changed the description

@jonrob for the RICH monitors/checkers I don't know which are monitors and which are checkers, could you help with that? For this issue, the main point of splitting them is that any algorithms running in the pit have a high priority to be thread-safe. (If they are already, feel free to tick them off)

For the RICH it is quite simple. Monitors (MC free) are here

https://gitlab.cern.ch/lhcb/Rec/-/tree/master/Rich/RichFutureRecMonitors/src

and the MC checkers are here

https://gitlab.cern.ch/lhcb/Rec/-/tree/master/Rich/RichFutureRecCheckers/src

your list above is not correct for the RICH, in that it is missing all the checks. How did you generate it ?

Also note, RICH monitors/checkers are 'brute force' thread safe in that I imply a mutex lock in the operator to do this. But, yes, moving to the new hist implementations is something to be done.

For this issue, I'm only looking at the histogram implementations, but of course the rest of the code needs to be thread-safe as well ;)

I took the list from https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/RecoConf/python/RecoConf/rich_data_monitoring.py After having a closer look, I see that not all of the monitors are running when doing do_data_monitoring=True, so I've reduced them a bit. You know better than me which monitors need to be running during data taking; this is just my minimal list.

Does it make sense that I see no extra monitoring/checkers algorithm when running on MC with do_mc_checking=True in the standalone_hlt2_reco?

That file only lists the data monitors. To see what checkers should be running please look at https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/RecoConf/python/RecoConf/rich_mc_checking.py as well.

For your second question, yes, if you are running with do_mc_checking=True then yes, I would hope that setting is being properly propagaged to where the RICH MC checking is enabled. I am not really an expert on how Moore configures itself, but from the looks of it this is supposed to happen here

https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/RecoConf/python/RecoConf/rich_add_reconstruction_monitoring_checking.py#L117

can you check if do_mc_checking=True is indeed being passed here ?

I cannot back find the output from the checkers when running standalone_hlt2_reco. The list of RICH folders is long though, so it'd be good to get an extra pair of eyes to check if I'm missing anything.

For example, for our dashboard jobs we're running with this options file: https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/RecoConf/options/hlt2_reco_baseline_with_mcchecking.py, so with do_mc_checking=True. You can download an output root file from the dashboard https://lblhcbpr.cern.ch/dashboards/reconstruction; first click another tab than "Track efficiencies"; then you can simply click on File2 to download a file named hlt2_reco_baseline_DC.

I couldv'e just given you a direct link to a file; here is one from today: https://lblhcbpr.cern.ch/root/186098/hlt2_reco_baseline_DC.root

changed the description

@sklaver Has there been any updates for this? I guess TrackMonitor and TrackVertexMonitor can be ticked as "done"?

mentioned in merge request !2710 (merged)

marked the checklist item TrackVertexMonitor as completed

marked the checklist item TrackVertexMonitor as incomplete

marked the checklist item TrackVertexMonitor as completed

marked the checklist item TrackMonitor as completed

How is this progressing? Are there other monitors that are now converted to the thread-safe histograms? @decianm @jonrob @valukash @ausachov.

Depends what you mean. As I have already said all the RICH monitors are already thread safe due to the explicit use of mutex locks. Converting these to the new thread safe implementations is somewhere on the todo list, but honestly at this point given everything else that needs to be done its not a high priority.

We converted the remaining track monitors to the functional framework and new histograms sometime ago but we had some issues. For now we just use mutex locks to avoid those issues but I have some time to try and fix the outstanding issues using the new thread safe implementations.

@jcobbled Sorry for being late in the reply. What "remaining track monitors" do you mean?

@decianm TrackFitMatchMonitor and TrackPV2HalfAlignMonitor and also perhaps TrackVPOverlapMonitor since I know this is used sometimes.

Ok. So what is here, here and here is thread-safe via mutex? I cannot really tell from the code.

And do you remember what the problems where when moving to the new histogram implementations?

I apply the mutex in my local version of the stack. Regarding the problems, this was a while ago, but from what I remember I had issues with the histograms types (int, float etc.) and some of the logic (for TrackPV2HalfAlignMonitor I think). I will take a look at these monitors again this week

@decianm, I have opened MRs for TRackPV2HalfAlignMonitor and TrackFitMatchMonitor with thread-safe histograms. I am currently testing them. I will also start working onTrackVPOverlapMonitor.

@sklaver do you know if there are some references I can check the plots against? These monitors don't tend to be used in Alignment.

@jcobbled an example to run alignment with monitoring (TrackFitMatchMonitor) is here https://gitlab.cern.ch/lhcb/Alignment/-/blob/master/Alignment/Humboldt/options/AlignVPHalvesModules.py#L106. The corresponding command is here https://gitlab.cern.ch/lhcb/Alignment/-/tree/master/Alignment/Humboldt. Similarly, you can add more monitors into the configuration. If you see that the upgraded monitor shows the same result as the old one then it should be fine. I hope it helps:)

@ausachov. Thanks, I also found these options and have tested TrackFitMatchMonitor and it seems fine (same histograms, entries etc.). I need to test TrackPV2HalfAlignMonitor in a similar way.

mentioned in merge request !2840 (merged)

marked the checklist item TrackFitMatchMonitor as completed

changed the description

mentioned in merge request !3074 (merged)

marked the checklist item PrFTHitsChecker as completed

marked the checklist item PrUTHitsChecker as completed

marked the checklist item PrUTHitsChecker as incomplete