Multithreading: Separate Thread Initialization from Run()
I have experienced issues in running few events with DepositionGeant4
in a multithreaded environment. Among other issues I ran into stack traces like the following (abbreviated):
-------- EEEE ------- G4Exception-START -------- EEEE -------
*** G4Exception : InvalidCondition
issued by : ParticlesWorspacePool::CreateWorkspace
Cannot create workspace twice for the same thread.
*** Fatal Exception *** core dump ***
G4WT42 > **** Step information is not available at this moment
G4WT41 > **** Step information is not available at this moment
G4WT40 >
-------- EEEE -------- G4Exception-END --------- EEEE -------
G4WT42 > *** G4Exception: Aborting execution ***
Thread 13 "allpix" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffc0aa6700 (LWP 13389)]
0x00007fffe088f387 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007fffe088f387 in raise () from /lib64/libc.so.6
#1 0x00007fffe0890a78 in abort () from /lib64/libc.so.6
#2 0x00007fffe7a60093 in G4Exception(char const*, char const*, G4ExceptionSeverity, char const*) () from /cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc8-opt/lib64/libG4global.so
#3 0x00007fffeefce564 in G4WorkerThread::BuildGeometryAndPhysicsVector() () from /cvmfs/sft.cern.ch/lcg/views/LCG_97python3/x86_64-centos7-gcc8-opt/lib64/libG4run.so
#4 0x00007fffe61c6008 in allpix::WorkerRunManager::GetNewInstanceForThread () at /data/simonspa/allpix-squared/src/tools/geant4/WorkerRunManager.cpp:188
#5 0x00007fffe61c4a58 in allpix::MTRunManager::InitializeForThread (this=this@entry=0x25847b0) at /data/simonspa/allpix-squared/src/tools/geant4/MTRunManager.cpp:86
#6 0x00007ffff7313040 in allpix::DepositionGeant4Module::run (this=0x255e800, event=0x7fff5c0008d0) at /data/simonspa/allpix-squared/src/modules/DepositionGeant4/DepositionGeant4Module.cpp:292
I traced this back to tow (separate) issues:
Firstly, some unclean initialization of Geant4 worker threads, mostly appearing when number_of_events < workers
.
Therefore I decided to introduce another stage in the succession of module commands: initializeThread()
, the equivalent to finalizeThread()
processed before the run()
function. It can be used to properly initialize thread-local things (like our WorkerRunManagers) before events are generated and processed.
Secondly, I realized that the ThreadPool did not catch exceptions from the finalize
and initialize
functions provided to it. I changed this, so the full lifetime of the worker thread is now covered.