Thread finalisation problem with TBB changes

Hi all,

There's an open question about what to do with the updates to the TBB interface, which @ishapova already has a decent solution for in !1067 (merged)

This is a related topic, based on my attempt to just remove deprecated TBB components (!1065 (closed)), following @akraszna already identifying the important changes in !982 (merged)

Please note that in ThreadPoolSvc.cpp L107 the construction of a tbb::task_scheduler_init is (optionally) removed !982 (diffs)

In testing I found that regardless of what value you set for the maximum parallelism in tbb::global_control, the TBB scheduler will now refuse to create more threads than there are physical cores (task_scheduler_init was used to override this).

This causes Gaudi to hang at the synchronisation barrier when finalising threads. The ThreadPoolSvc assumes that there are a known number of threads that must be finalised, and launches that many tasks to do so. Each task waits at the synchronisation barrier. If there are actually fewer threads, Gaudi waits indefinitely for the tasks that have been enqueued but have no thread to run in.

How should this be addressed? As I see it there are three options:

1. This behaviour will somehow get fixed as part of a more complete update to the TBB usage in Gaudi, don't worry for now.
2. The thread finalisation behaviour doesn't really do anything anyway (there is the option to run a list of tools, but I don't think that's ever used). So, we should just remove this entirely.
3. Finalise threads in an ad-hoc manner, using the destructor of something in Thread Local Storage maybe? Or perhaps investigate https://www.threadingbuildingblocks.org/docs/help/reference/task_scheduler/task_scheduler_observer.html

I think the TBB intention is that we should not be able to find out how many threads the scheduler is using, so the current approach cannot/should not be fixed.

On a related note, how does it affect our performance studies if the scheduler cannot be made to oversubscribe the CPU with Nthreads > Ncores?

Cheers, Ben