singular matrix in TrackBestTrackCreator with VeloClusterTrackingSIMDFull

I did a small investigation. The error occurs here: https://gitlab.cern.ch/lhcb/Rec/-/blob/master/Tr/TrackFitEvent/src/Measurement.cpp#L126

called from here:

https://gitlab.cern.ch/lhcb/Rec/-/blob/master/Tr/TrackFitter/src/TrackMasterFitter.cpp#L557

Looking at the LHCbIDs and positions (x/y/z) from the nodes I get:

{ LHCbID : {  VPChannelID : 23240186 : row = 250 col = 157 chip = 2 sensor = 88 } }
{ LHCbID : {  VPChannelID : 25336798 : row = 222 col = 155 chip = 2 sensor = 96 } }
{ LHCbID : {  VPChannelID : 27433409 : row = 193 col = 153 chip = 2 sensor = 104 } }
{ LHCbID : {  VPChannelID : 29530277 : row = 165 col = 152 chip = 2 sensor = 112 } }
{ LHCbID : {  VPChannelID : 31626889 : row = 137 col = 150 chip = 2 sensor = 120 } }
{ LHCbID : {  VPChannelID : 33723501 : row = 109 col = 148 chip = 2 sensor = 128 } }
{ LHCbID : {  VPChannelID : 35820368 : row = 80 col = 147 chip = 2 sensor = 136 } }
{ LHCbID : {  VPChannelID : 37916980 : row = 52 col = 145 chip = 2 sensor = 144 } }
{ LHCbID : {  VPChannelID : 40132600 : row = 248 col = 95 chip = 0 sensor = 153 } }
{ LHCbID : {  VPChannelID : 42230947 : row = 163 col = 100 chip = 0 sensor = 161 } }
{ LHCbID : {  VPChannelID : 44329522 : row = 50 col = 106 chip = 0 sensor = 169 } }
{ LHCbID : { FTChannelID :  channel =40 sipm =1 mat =2 module=0 quarter=0 layer=0 station=1 } }
{ LHCbID : { FTChannelID :  channel =89 sipm =1 mat =1 module=0 quarter=0 layer=1 station=1 } }
{ LHCbID : { FTChannelID :  channel =42 sipm =0 mat =2 module=0 quarter=0 layer=2 station=1 } }
{ LHCbID : { FTChannelID :  channel =90 sipm =2 mat =1 module=0 quarter=0 layer=3 station=1 } }
{ LHCbID : { FTChannelID :  channel =28 sipm =1 mat =2 module=0 quarter=0 layer=0 station=2 } }
{ LHCbID : { FTChannelID :  channel =87 sipm =1 mat =1 module=0 quarter=0 layer=1 station=2 } }
{ LHCbID : { FTChannelID :  channel =21 sipm =0 mat =2 module=0 quarter=0 layer=2 station=2 } }
{ LHCbID : { FTChannelID :  channel =101 sipm =2 mat =1 module=0 quarter=0 layer=3 station=2 } }
{ LHCbID : { FTChannelID :  channel =19 sipm =1 mat =2 module=0 quarter=0 layer=0 station=3 } }
{ LHCbID : { FTChannelID :  channel =86 sipm =1 mat =1 module=0 quarter=0 layer=1 station=3 } }
{ LHCbID : { FTChannelID :  channel =3 sipm =0 mat =2 module=0 quarter=0 layer=2 station=3 } }
{ LHCbID : { FTChannelID :  channel =110 sipm =2 mat =1 module=0 quarter=0 layer=3 station=3 } }
{ LHCbID : { UT UTChannelID : 19668470 : type=2 strip=502 sector=7 detRegion=2 layer=1 station=1 } }
{ LHCbID : { UT UTChannelID : 20209743 : type=2 strip=79 sector=24 detRegion=2 layer=2 station=1 } }
{ LHCbID : { UT UTChannelID : 21765399 : type=2 strip=279 sector=7 detRegion=2 layer=1 station=2 } }
{ LHCbID : { UT UTChannelID : 22289775 : type=2 strip=367 sector=7 detRegion=2 layer=2 station=2 } }
-233.136 -550.792 11900
-227.752 -437.627 9450
-227.752 -437.627 9450
-227.627 -435.451 9402.89
-227.437 -431.981 9333.12
-227.244 -428.74 9262.87
-227.048 -425.52 9193.1
-225.599 -403.592 8717.89
-225.366 -400.371 8648.12
-225.126 -397.129 8577.87
-224.88 -393.908 8508.1
-223.012 -372.108 8035.89
-222.698 -368.886 7966.11
-222.37 -365.642 7895.87
-222.032 -362.42 7826.1
-220.259 -347.358 7500
-108.238 -123.2 2647.84
-106.047 -120.658 2592.84
-97.2336 -110.488 2372.84
-2.42953e+06 3.05622e+08 2332.84
9.09203e+23 -3.75479e+19 2165
-3.80378e+20 2.93381e+16 990
-2.13454e+11 -5.93429e+11 770
2.45938e+21 -2958.11 488.081
-19.9276 -22.7323 488.081
-166.738 -18.0781 388.081
-16.0782 -18.0781 388.081
-12.9745 -14.5851 313.081
-12.9745 -14.5851 313.081
-10.8567 -12.202 261.919
-10.8567 -12.202 261.919
-9.82176 -11.0374 236.919
-9.82176 -11.0374 236.919
-8.78674 -9.87266 211.919
-8.78674 -9.87266 211.919
-7.75167 -8.7079 186.919
-7.75167 -8.7079 186.919
-6.71655 -7.54311 161.919
-6.71655 -7.54311 161.919
-5.68138 -6.37828 136.919
-5.68138 -6.37828 136.919
-4.64615 -5.21343 111.919
-4.64615 -5.21343 111.919
-3.61087 -4.04857 86.919
-3.61085 -4.04857 86.919

Apart from the obviously nonsensical numbers, also this:

-166.738 -18.0781 388.081
-16.0782 -18.0781 388.081

looks a bit odd to me.

@ahennequ

It is indeed quite stange. Are you sure about the code that prints the positions ? I count 11 VP lhcbids but twice as many positions (that would explain why most hits seems duplicated). Understanding what exactly is printed would help to understand the issue.

Ah I answered my question myself: the positions are for the nodes (separated X/Y), so I'd look at the code that makes the measurements.

yes, indeed, this is twice per LHCbID for the VP.

The position for the X and Y measurements should be identical, so clearly something is going wrong in the copy leading to -16.0782 becoming -166.738.

I did some more investigations:

The creation of the measurements itself seems fine, there the coordinates are identical.
Looking at the reference vector of the nodes before the first fit iteration (i.e. before here) shows an odd pattern. The track is essentially straight in x in the Velo at -201.0, but seems to make odd jumps after the Velo:

TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters : -71.523, -79.5258, -0.0166208, -0.017402, 3.08435e-05 z :	2165 }) 
TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters : -40.4811, -45.5426, -0.0370215, -0.0414278, 3.08435e-05 z :	990 }) 
TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters : -31.8602, -35.8652, -0.0413781, -0.0465838, 3.08435e-05 z :	770 }) 
TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters : -201.659, -22.7323, -0.00292889, -0.0465838, 3.08435e-05 z : 488.081}) 
TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters : -201.659, -22.7323, -0.00292889, -0.0465838, 3.08435e-05 z : 488.081 })
TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters : -201.366, -18.0739, -0.00292889, -0.0465838, 3.08435e-05 z : 388.081 }) 
TrackBestTrackCreator.TrackMasterFitter  INFO  ref state = ({ parameters :  -201.366, -18.0739, -0.00292889, -0.0465838, 3.08435e-05 z : 388.081 })

The states at 770 and 990 do not correspond to a measurement, so I wonder if anything goes wrong in their creation.

770 is the endvelo state, so you could check what's its position from the velo fit

At the end of the Velo tracking, the endvelo state is:

-31.8602 -35.8652 -0.0413781 -0.0465838 770

so exactly like what I saw above for this state.

@freiss I saw that you run both versions, the default and the full version of the Velo tracking in your script. Is there a reason for this?

The state is wrong, it should be calculated here: https://gitlab.cern.ch/lhcb/Rec/-/blob/master/Pr/PrPixel/src/VeloClusterTrackingSIMD.cpp#L922

It is a simple straight line extrapolation based on the last 2 hits of the tracks, so it should be:

x: -201.659 + (770 - 488.081) * -0.00292889 = -202.48470974
y: -22.7323 + (770 - 488.081) * -0.0465838 = -35.8651583122 
tx: -0.00292889
ty: -0.0465838

y and ty are correct, but x and tx are wrong. Could you print the value of the state before it's added to the container and the x0 and tx values used to compute it ?

yes, I tried out both version to be sure that the problem only appears with the full version

The numbers before putting it in the container are:

x0: 0.000906944 tx: -0.0413781 y0: 0.00436735 ty: -0.0465838

and the state:

-31.8602 -35.8652  770 -0.0413781 -0.0465838

The code is just:

info() << "x0: " << x0 << " tx: " << tx << " y0: " << y0 << " ty: " << ty << endmsg;
info() <<  x0 + StateParameters::ZEndVelo * tx << " " <<  y0 + StateParameters::ZEndVelo * ty << " " << StateParameters::ZEndVelo << " " << tx << " " << ty << endmsg;

And looking at the ClosestToBeamState in the VeloKalman, I get:

x: -11.7204 y: -6.75769 tx: 0.00104267 ty: -0.00180839 z:9547.43

After some investigations, it turns out it's a config issue, you forgot to bind also the hits:

with make_hlt2_tracks.bind(fast_reco=True), make_VeloClusterTrackingSIMD_tracks.bind(make_tracks=VeloClusterTrackingSIMDFull), make_VeloClusterTrackingSIMD_hits.bind(make_tracks=VeloClusterTrackingSIMDFull), reco_prefilters.bind(gec=False):
    run_reconstruction(options, standalone_hlt2_reco)

So the algorithm is running twice, with one version producing the hits and the other the tracks, making them out-of-sync.

@ahennequ @decianm thanks a lot for looking into this. Indeed binding also the hits works.
Sorry for not having spotted that I was running both in the sequence earlier. I only now understand that was what Michel's question was getting at.
This is beyond this issue, but to me this seems to be an easy mistake to make. Is there a reason why it is allowed to create the tracks and hits with different algorithms?

Interesting question. It's a byproduct of 2 things:

The functional framework requires dependencies to be explicit
Hits are separated of tracks since they could be used for other algorithms (not all hits are used)

Note that for all others detectors the hits and tracks are produced by different algorithms, but the problem remains the same, once a track is created with a hit container the same hit container should be used with this track.

We could keep a pointer to the hits inside the velo container (and we actually have new mechanisms to do it properly with soacollections) but that would mean the framework will not know about the dependency (I'll let @sponce and @graven comment on that).

Simply issuing a warning might be enough (if that is possible in the configuration).

note that yes, 'data dependencies' must be explicit because it influences scheduling, i.e. the framework wants to make sure that all declared inputs are made available prior to running the component which declares them, i.e. it wants to know the data dependencies of the components it schedules.

But another point is that 'once data is published into the TES, it shall be considered immutable'. This implies that it is perfectly fine to produce 'views' and store them in the TES, provided these views refer to data which either resides in the TES, or which will reside in the TES by the time the view is made accessible from the TES (the latter typically requires the data underlying the view to be 'movable', i.e. moving it into the TES does not invalidate the corresponding view). So a component can have a data dependency on the view, as it can assume that once a view is published, implicitly the underlying source was also published, as that is the responsibility of the producer of the view (which eg. could have stated on explicit dependency on the underlying source, taking it from the TES, or it produced the underlying source 'alongside' the view, and properly moved it into the TES). So provided that the above small print holds, there is no reason why one couldn't have a pointer to the hits inside the velo container...

Note that currently the framework does not actively track data dependencies amongst data objects -- which does affect the possibility to persist 'views'. If those views need to be persisted, then the persistency code must be able to deal with them in some way... eg. it must either somehow be able to figure out what the actual data is, make sure that (or the relevant subset) is also persisted, and then provide a means to re-create the view on read back, or employ some other mechanism depending on the specific case to 'make it work'...

would it be possible to have a helper function like

@configurable
wrap_VeloClusterTrackingSIMD(make_tracks=VeloClusterTrackingSIMD) {
  return make_tracks
}

which is then called by make_VeloClusterTrackingSIMD_tracks and make_VeloClusterTrackingSIMD_hits, so that one would only have to remember the binding of the wrapper function?

@freiss Indeed, I guess one can essentially make a (I slightly modifying your example)

@configurable
make_VeloClusterTrackingSIMD(algorithm = VeloClusterTrackingSIMD)
   return algorithm

I made an MR here: Moore!1432 (merged)

mentioned in merge request Moore!1432 (merged)

closed with commit 5bf9c955

mentioned in commit Moore@5bf9c955

singular matrix in TrackBestTrackCreator with VeloClusterTrackingSIMDFull

Designs

Child items 0

Activity