scaling issue with test_hlt2_all_lines test
I’ve been noticing some issues with the test_hlt2_all_lines test in Moore over the last few weeks, that I think are starting to point towards some scaling issues in Moore, that it might be pointing at.
Basically, I am seeing it time out in a number of places. see e.g. the clang debug build
there it times out after 2669.22
there is also a stack trace that may or may not point at something useful ?
Even in the gcc debug build, where it completes OK, its taking a very long time. 1825.82 secs, so only just making it really
finally, the absolute worst case is the address sanitiser builds. There I am seeing it take down the entire test build after 10 hours. see
https://jenkins-lhcb-nightlies.web.cern.ch/job/nightly-builds/job/tests/2139777//console
which is currently ongoing, but I know will eventually bomb out based on the last few days experience. I suspect something truly pathological is happening there (perhaps there is a crash / seg. fault we don’t get to see). the address sanitisers run slower than the regular build, but not that slow.
Anyway. I think any this points to a potential scaling issue in Moore, that might just get worse and worse as we get more and more lines (we had something like this before if you recall Alex, when I added the RICH reco to HLT2, due to the large increase in algorithms this implied) so I think we should start to think about it, before it does get much worse.