A large fraction of failure in MC truth-matching when using Retina Clusters in Velo tracking reconstruction
As discussed in mattermost channel
- The reconstructed kinematics when using VeloSP or Retina clusters match each other very well
- but quite large fraction of reconstructed tracks can not be matched when using Retina Clusters
2023 MC sample used: root://x509up_u128460@eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/MC/Dev/DIGI/00194513/0000/00194513_00000005_1.digi
Also checked with 2022 minbias MC: root://eoslhcb.cern.ch//eos/lhcb/grid/prod/lhcb/MC/Upgrade/XDIGI/00171960/0000/00171960_00006077_1.xdigi
See more plots in attachment. truthmatch_20231007.pdf
The PrTrackRecoDumper algorithm was used to dump the reconstructed info,
with this option: /afs/cern.ch/user/p/peilian/eoslhcb/public/forGiovanni/hlt2_trackrecodumper.py
Designs
Related merge requests 7
When these merge requests are accepted, this issue will be closed automatically.
- lhcb-datapkg/AnalysisProductions !1041
- lhcb-datapkg/AnalysisProductions !746
- lhcb-datapkg/AnalysisProductions !715
Activity
Some of the metrics could depend on what is considered reconstructible, to have a close look…
Edited by Laurent DufourThe ghost rates reported in the reference file for the current tests are very suspicious.
That test is as far as I can tell currently disabled in the nightlies although I agree the reference is very suspicious. Given how many tests there are probably easiest if @decianm tells us which one is testing the production sequence.
Thanks @gligorov for pointing out that this commit: https://gitlab.cern.ch/lhcb/Moore/-/blob/3e0ff230753dbfc1807fc700f1090ca314aee845/Hlt/RecoConf/tests/refs/hlt2_reco_retina_with_mcchecking.ref.detdesc
introduced the 0% efficiency. This ref update was to update to the "newer" samples with more data-like decoding versions Moore!1703 (merged) (and related MRs).
However, this does not (necessarily) explain what @peilian sees, as the samples were produced independently, so I think we need @gbassi input.
Note that there are other tests which are fine e.g. https://gitlab.cern.ch/lhcb/Moore/-/blob/master/Hlt/RecoConf/tests/refs/hlt1_reco_retina_with_mcchecking_FTv6.ref.detdesc
assigned to @gbassi
Note that, after fixing, we probably have to reproduce the HLT2-processed MC samples for 2022, and it has to be included for the 2023 production as well.
Edited by Laurent Dufour
mentioned in merge request lhcb-datapkg/AnalysisProductions!715 (merged)
Some insights: The file
default_input_and_conds_hlt2.pywhich is used by many tests (e.g.hlt2_light_reco_pr_kf_without_UT_with_mcchecking.qmt) includes:# Use velo retina decoding: make_VeloClusterTrackingSIMD.global_bind( algorithm=VeloRetinaClusterTrackingSIMD) make_velo_full_clusters.global_bind( make_full_cluster=VPRetinaFullClusterDecoder)i.e. all tests using these inputs will use Retina clusters, even if they are not labelled so.
Running a comparison with and without these lines results in:
RETINA: VeloTrackChecker_26ed5b53 INFO Results VeloTrackChecker_26ed5b53 INFO **** Velo 17851 tracks including 475 ghosts [ 2.66 %], Event average 3.32 % **** VeloTrackChecker_26ed5b53 INFO 01_velo : 7588 from 7722 [ 98.26 %] 111 clones [ 1.44 %], purity: 99.75 %, hitEff: 95.71 %, hitEffFirst3: 95.99 %, hitEffLast: 95.46 % VeloTrackChecker_26ed5b53 INFO 02_long : 4357 from 4381 [ 99.45 %] 40 clones [ 0.91 %], purity: 99.83 %, hitEff: 96.64 %, hitEffFirst3: 96.96 %, hitEffLast: 96.41 % VeloTrackChecker_26ed5b53 INFO 03_long_P>5GeV : 2784 from 2790 [ 99.78 %] 18 clones [ 0.64 %], purity: 99.86 %, hitEff: 97.00 %, hitEffFirst3: 97.44 %, hitEffLast: 96.75 % VeloTrackChecker_26ed5b53 INFO 04_long_strange : 222 from 234 [ 94.87 %] 1 clones [ 0.45 %], purity: 99.12 %, hitEff: 96.50 %, hitEffFirst3: 97.16 %, hitEffLast: 95.84 % VeloTrackChecker_26ed5b53 INFO 05_long_strange_P>5GeV : 109 from 113 [ 96.46 %] 0 clones [ 0.00 %], purity: 99.43 %, hitEff: 97.25 %, hitEffFirst3: 97.55 %, hitEffLast: 97.15 % VeloTrackChecker_26ed5b53 INFO 06_long_fromD : 22 from 22 [100.00 %] 0 clones [ 0.00 %], purity:100.00 %, hitEff: 97.86 %, hitEffFirst3: 98.48 %, hitEffLast: 97.64 % VeloTrackChecker_26ed5b53 INFO 07_long_fromD_P>5GeV : 7 from 7 [100.00 %] 0 clones [ 0.00 %], purity:100.00 %, hitEff: 97.55 %, hitEffFirst3:100.00 %, hitEffLast: 96.66 % VeloTrackChecker_26ed5b53 INFO 08_long_electrons : 267 from 282 [ 94.68 %] 4 clones [ 1.48 %], purity: 98.24 %, hitEff: 90.09 %, hitEffFirst3: 89.18 %, hitEffLast: 89.30 % VeloTrackChecker_26ed5b53 INFO 11_long_fromD_P>3GeV_Pt>0.5GeV : 7 from 7 [100.00 %] 0 clones [ 0.00 %], purity:100.00 %, hitEff: 96.12 %, hitEffFirst3:100.00 %, hitEffLast: 94.62 % VeloTrackChecker_26ed5b53 INFO 11_long_strange_P>3GeV_Pt>0.5GeV : 39 from 44 [ 88.64 %] 0 clones [ 0.00 %], purity: 99.57 %, hitEff: 97.99 %, hitEffFirst3: 96.58 %, hitEffLast: 99.42 % with SUPERPIXEL: VeloTrackChecker_3889b1d5 INFO Results VeloTrackChecker_3889b1d5 INFO **** Velo 17892 tracks including 480 ghosts [ 2.68 %], Event average 3.41 % **** VeloTrackChecker_3889b1d5 INFO 01_velo : 7585 from 7722 [ 98.23 %] 125 clones [ 1.62 %], purity: 99.78 %, hitEff: 95.69 %, hitEffFirst3: 95.81 %, hitEffLast: 95.31 % VeloTrackChecker_3889b1d5 INFO 02_long : 4356 from 4381 [ 99.43 %] 48 clones [ 1.09 %], purity: 99.85 %, hitEff: 96.64 %, hitEffFirst3: 96.72 %, hitEffLast: 96.53 % VeloTrackChecker_3889b1d5 INFO 03_long_P>5GeV : 2786 from 2790 [ 99.86 %] 22 clones [ 0.78 %], purity: 99.88 %, hitEff: 97.05 %, hitEffFirst3: 97.26 %, hitEffLast: 96.96 % VeloTrackChecker_3889b1d5 INFO 04_long_strange : 223 from 234 [ 95.30 %] 2 clones [ 0.89 %], purity: 99.32 %, hitEff: 96.28 %, hitEffFirst3: 97.04 %, hitEffLast: 95.31 % VeloTrackChecker_3889b1d5 INFO 05_long_strange_P>5GeV : 110 from 113 [ 97.35 %] 0 clones [ 0.00 %], purity: 99.44 %, hitEff: 97.39 %, hitEffFirst3: 97.88 %, hitEffLast: 96.68 % VeloTrackChecker_3889b1d5 INFO 06_long_fromD : 22 from 22 [100.00 %] 0 clones [ 0.00 %], purity:100.00 %, hitEff: 97.86 %, hitEffFirst3:100.00 %, hitEffLast: 96.99 % VeloTrackChecker_3889b1d5 INFO 07_long_fromD_P>5GeV : 7 from 7 [100.00 %] 0 clones [ 0.00 %], purity:100.00 %, hitEff: 97.55 %, hitEffFirst3:100.00 %, hitEffLast: 96.66 % VeloTrackChecker_3889b1d5 INFO 08_long_electrons : 271 from 282 [ 96.10 %] 3 clones [ 1.09 %], purity: 98.10 %, hitEff: 92.10 %, hitEffFirst3: 88.20 %, hitEffLast: 94.02 % VeloTrackChecker_3889b1d5 INFO 11_long_fromD_P>3GeV_Pt>0.5GeV : 7 from 7 [100.00 %] 0 clones [ 0.00 %], purity:100.00 %, hitEff: 96.12 %, hitEffFirst3:100.00 %, hitEffLast: 94.62 % VeloTrackChecker_3889b1d5 INFO 11_long_strange_P>3GeV_Pt>0.5GeV : 40 from 44 [ 90.91 %] 0 clones [ 0.00 %], purity: 99.58 %, hitEff: 98.04 %, hitEffFirst3: 97.50 %, hitEffLast: 99.20 %The numbers are very close, so I don't think there is an issue here.
@peilian In your dumper code I see:
VPLightClusterLocation= make_VPClus_hits()Can you try to replace this with
VPLightClusterLocation=make_RetinaClustersEdited by Michel De Cian
Thanks @decianm for the point. This doesn't matter for the information I used in the plots as this is only used to get the position of hits but we don't use them for anything else.
In any case, I just changed to
make_RetinaClustersaccordingly, and no changes seen in the plots I showed before as expected.Edited by Peilian LiIn addition, can you do:
# Use velo retina decoding: make_VeloClusterTrackingSIMD.global_bind( algorithm=VeloRetinaClusterTrackingSIMD) make_velo_full_clusters.global_bind( make_full_cluster=VPRetinaFullClusterDecoder)?
I see:
make_links_lhcbids_mcparticles_tracking_system()but this does not know that it should use Retina either
I would opt for making Retina clusters the default in the reconstruction, and accordingly in all the options, as it's clearly not trivial to get the options correct. If there are some tests still using old MC samples where Retina clusters were not available, then only there use binds to use VeloSP in the options (and replace them asap with newer samples)
-
As @ldufour commented, this problem was in all the generated 2022 DST MC and should be fixed accordingly.
Edited by Peilian Li -
All as in MC productions for which HLT2 was run centrally, digi files should be ok. Given we will have the HLT2 reprocessing of 2022 data and will need to run on MC with the respective Moore version as well, I'd fix this issue directly for those productions. But I agree we should not forget about it
It is followed here:
lhcb-simulation/lbmcsubmit#69 (closed) lhcb-simulation/lbmcsubmit#68
Should we introduce a
default_vp_light_clusters( vp_light_cluster_creator=... )function that is overwritten by the bind in case a non-default configuration is used? There are more places where we work with VPLightClusters, which typically one all changes together when switching between SP and Retina clusters.(note that this does not remove the flexibility to use specific binds at all) (and actually a similar functionality is then also introduced for vp_full_clusters)
Edited by Laurent DufourWhich exact bind is the one that solved the MC matching issue? The one for
make_links_lhcbids_mcparticles_tracking_system()? I'm a bit puzzled this is needed since for the Allen velo tracking for example it is not the case (see allen_gaudi_velo.pyAnother difference I noted to the Allen velo tracking checking scripts is the Retina decoding. Above you pointed to
make_velo_full_clusters.global_bind( make_full_cluster=VPRetinaFullClusterDecoder)while
make_velo_full_clusters.bind(make_full_cluster=VPRetinaFullClustering)is used in Allen. Is this an overkill, since it does a CPU emulation of the Retina clustering rather than just decoding it @gbassi ?
What is your idea for a
default_vp_light_clusters( vp_light_cluster_creator=... )function @ldufour ? To set all the required binds? Then it would probably make sense to have one for retina clusters (the default) and one for super pixels that can be used as alternative?Hi @dovombru, for newer MC samples (Retina decoding version 4)
VPRetinaFullClusteringis definitely an overkill because it produces full clusters out of SuperPixels instead of directly decoding the Retina cluster raw bank into full clusters. HoweverVPRetinaFullClusteringis still needed for older samples (Retina decoding versions 2 and 3), where it is not possible to decode cluster words into full clusters, so full clusters need to be recomputed from SuperPixels.Hi, as reported in the messages above, to use
VPRetinaClusterbothmake_VeloClusterTrackingSIMDandmake_velo_full_clustersmust be bound:make_VeloClusterTrackingSIMD.bind( algorithm=VeloRetinaClusterTrackingSIMD) make_velo_full_clusters.bind( make_full_cluster=VPRetinaFullClusterDecoder)Performing only one of the two bind, tracks reconstructed using a kind of clusters are matched to MC-tracks through a different kind of clusters. Technically it is not a bug of the code, but a wrong setup of the custom sequence.
As it is easy to forget to perform the second bind, I opened the MR Moore!2786 (merged) to introduce a new function that with a single bind set the two algorithms.
I do not know if we want to close this issue, or if we want to wait the MR to be merged.
please note Moore!2166 (merged) which cleans up the need for some of these binds... so please first merge Moore!2166 (merged) (and dependent MR) and then see what is still needed.
mentioned in merge request lhcb-datapkg/AnalysisProductions!746 (merged)
unassigned @gbassi
mentioned in merge request Moore!2786 (merged)
added MC checking RTA Tracking labels
assigned to @dovombru
closed with merge request Moore!2786 (merged)
mentioned in commit Moore@0fb2e4d4
I'm reopening this because
- We need to make sure it's properly fixed and tested in master
- We need to backport to the
2023-hlt2pp-patchesbranch used for 2023 MC production.
@sesen do you know if this is properly tested in master?
-
/cc @sberneta
-
@rmatev can you point out which options are to be used for running on MC productions?
@flazzari since you prepared the simplification in Moore!2786 (merged) would you be able to help to propagate to
2023-hlt2pp-patchesand test the MC options to make sure the matching is properly using the Retina clusters? I missed something: this is not a code issue, but a misconfiguration of a private option file. At time, all the option files in Moore was correct. I deployed a simplification that require a single bind for using RetinaClusters, but it's still possible to use the old way with two binds. I can help to propagate the simplification, but basically it consist to apply the commit to
2023-hlt2pp-patches-
sorry, @flazzari I was probably not clear. This last discussion is in the context of providing options to run HLT2 centrally on MC. We need to make sure those options use the proper configuration, it might be the case already. On top of this, we would like to have a test to ensure this doesn't break with further developments.
-
@flazzari can you please add the necessary bind to https://gitlab.cern.ch/lhcb/Moore/-/blob/v54r16p4/Hlt/Hlt2Conf/python/Hlt2Conf/settings/hlt2_pp_commissioning.py?ref_type=tags#L564 and make a MR targeting the
2023-hlt2pp-patchesbranch? We need this for the 2023 MC productions with HLT2 running centrally. @cmarinbe See Moore!3154 (merged) . However is not well clear to me if this function calls the MC-truth matching, and therefore if the second bind is required.
Edited by Federico Lazzari
mentioned in merge request Moore!3154 (merged)
mentioned in merge request lhcb-datapkg/AnalysisProductions!1041 (merged)
