New servers not using full CPU power

Online has installed the new HLT2 servers on IT5 and made them available in the HLT2 run control. These machines have 256 threads available. We have tested running HLT2 on one of the nodes and observe that while we use all the CPUs, most of their power is spent waiting:

@tcolombo and @flpisani have kindly run a profiler, the results of which can be found at: https://lbgroups.cern.ch/online/flamegraph_hlt5_256.html This indicates that the main problem is in the Kalman Filter calling futex too often.

@ahennequ @ausachov @graven is this something you can look into? We estimate this slows down the new servers by a factor of around 4.