`hlt1_pp_forward_then_matching_and_downstream_no_gec_validation` test in Allen-ci fails occassionally in geforcetx2080ti
After !1095 (merged) was merged, the hlt1_pp_forward_then_matching_and_downstream_no_gec_validation test fails from time to time on the geforcetx2080ti (for example https://gitlab.cern.ch/lhcb/Allen/-/jobs/35134099)
Crash log:
an illegal memory access was encountered (700) at ../backend/include/CUDABackend.h: 130
[625](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L625)Execution of algorithm downstream\_validator raised an exception
[626](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L626)terminate called after throwing an instance of 'std::invalid\_argument'
[627](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L627) what(): cudaCheck failed
[628](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L628)./scripts/ci/allen\_wrapper.sh: line 96: 1830125 Aborted (core dumped) CUDA\_DEVICE\_ORDER=PCI\_BUS\_ID CUDA\_VISIBLE\_DEVICES=0 numactl --cpunodebind=0 --membind=0 ./toolchain/wrapper ./Allen --params external/ParamFiles/ --sequence hlt1\_pp\_forward\_then\_matching\_and\_downstream\_no\_gec\_validation.json --mdf /scratch/allen\_data/mdf\_input/SciFiv6\_upgrade\_DC19\_01\_Bs2PhiPhiMD\_retinacluster\_v1\_newLHCbID\_v2\_new\_UT\_geometry.mdf -g /scratch/allen\_geometries/upgrade\_DC19\_01\_MinBiasMD\_new\_UT\_geometry -n 1000 -m 1100
[629](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L629)AllenCI ERROR Command './scripts/ci/allen\_wrapper.sh --params external/ParamFiles/ --sequence hlt1\_pp\_forward\_then\_matching\_and\_downstream\_no\_gec\_validation.json --mdf /scratch/allen\_data/mdf\_input/SciFiv6\_upgrade\_DC19\_01\_Bs2PhiPhiMD\_retinacluster\_v1\_newLHCbID\_v2\_new\_UT\_geometry.mdf -g /scratch/allen\_geometries/upgrade\_DC19\_01\_MinBiasMD\_new\_UT\_geometry -n 1000 -m 1100 2>&1' returned non-zero exit status 134.
[630](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L630)Traceback (most recent call last):
[631](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L631) File "/localdisk/gitlab-runs/R9xDbcs\_/0/lhcb/Allen/scripts/ci/run\_tests.py", line 260, in run\_allen\_test
[632](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L632) allen\_log = check\_output(
[633](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L633) File "/usr/lib64/python3.9/subprocess.py", line 424, in check\_output
[634](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L634) return run(\*popenargs, stdout=PIPE, timeout=timeout, check=True,
[635](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L635) File "/usr/lib64/python3.9/subprocess.py", line 528, in run
[636](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L636) raise CalledProcessError(retcode, process.args,
[637](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L637)subprocess.CalledProcessError: Command './scripts/ci/allen\_wrapper.sh --params external/ParamFiles/ --sequence hlt1\_pp\_forward\_then\_matching\_and\_downstream\_no\_gec\_validation.json --mdf /scratch/allen\_data/mdf\_input/SciFiv6\_upgrade\_DC19\_01\_Bs2PhiPhiMD\_retinacluster\_v1\_newLHCbID\_v2\_new\_UT\_geometry.mdf -g /scratch/allen\_geometries/upgrade\_DC19\_01\_MinBiasMD\_new\_UT\_geometry -n 1000 -m 1100 2>&1' returned non-zero exit status 134.
[638](https://gitlab.cern.ch/lhcb/Allen/-/jobs/34908105#L638)AllenCI FAIL Error during efficiency test {'type': 'efficiency', 'sequence': 'hlt1\_pp\_forward\_then\_matching\_and\_downstream\_no\_gec\_validation', 'dataset': 'SciFiv6\_upgrade\_DC19\_01\_Bs2PhiPhiMD\_retinacluster\_v1\_newLHCbID\_v2\_new\_UT\_geometry', 'geometry': 'upgrade\_DC19\_01\_MinBiasMD\_new\_UT\_geometry'}. See above for output and traceback.