SIGSEGV probably from threadpool
I just installed AP2 on one of my more powerful machines, it's the current master branch (5e333eca) + my two open MRs (!930 and !929 (merged)). First time running it, it SIGSEV-ed. Looks like some concurrency issue. I then attempted to run it single threaded - worked, six threads - worked, and ultimately on 31 again - worked.
Possibly a problem which is there all the time, but just likely to appear with many threads. I don't have many details, but since hunting down such issues can be a pain in the ass I though I'd provide you with what I have:
-bash-4.2$ /data/scratch/tbisanz/allpix-squared/bin/allpix -c simulation_full.conf
|12:51:36.678| (STATUS) Welcome to Allpix^2 v2.0.0-1791-ge8471132b
|12:51:36.679| (STATUS) Initialized PRNG with system entropy seed 7046976385323596384
|12:51:40.936| (STATUS) Loaded 7 modules
|12:51:40.936| (STATUS) Multithreading enabled, processing events in parallel on 31 worker threads
|12:51:40.936| (STATUS) Allocating a total of 7936 event slots for buffered modules
|12:51:40.936| (STATUS) Initializing 7 module instantiations
################################
!!! G4Backtrace is activated !!!
################################
|12:51:51.791| (STATUS) Initialized 7 module instantiations
### CAUGHT SIGNAL: 11 ### address: 0x10, signal = SIGSEGV, value = 11, description = segmentation violation. Address not mapped to object.
Backtrace:
[PID=2063, TID=-1][0/6]> /cvmfs/sft.cern.ch/lcg/releases/gcc/10.3.0-f5826/x86_64-centos7/lib64/libstdc++.so.6(_ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_+0x11a) [0x7f58d344d20a]
[PID=2063, TID=-1][1/6]> /data/scratch/tbisanz/allpix-squared/lib/libAllpixCore.so(_ZNSt8_Rb_treeINSt6thread2idESt4pairIKS1_jESt10_Select1stIS4_ESt4lessIS1_ESaIS4_EE22_M_emplace_hint_uniqueIJRKSt21piecewise_construct_tSt5tupleIJOS1_EESF_IJEEEEESt17_Rb_tree_iteratorIS4_ESt23_Rb_tree_const_iteratorIS4_EDpOT_+0x74) [0x7f58d4a7bbb4]
[PID=2063, TID=-1][2/6]> /data/scratch/tbisanz/allpix-squared/lib/libAllpixCore.so(_ZN6allpix10ThreadPool6workerEmRKSt8functionIFvvEES5_+0xbc) [0x7f58d4a79d0c]
[PID=2063, TID=-1][3/6]> /cvmfs/sft.cern.ch/lcg/releases/gcc/10.3.0-f5826/x86_64-centos7/lib64/libstdc++.so.6(+0xd6ff0) [0x7f58d345fff0]
[PID=2063, TID=-1][4/6]> /lib64/libpthread.so.0(+0x7ea5) [0x7f58d4ae7ea5]
[PID=2063, TID=-1][5/6]> /lib64/libc.so.6(clone+0x6d) [0x7f58d2d9cb0d]
: Segmentation fault (Address not mapped to object [0x10])
Aborted (core dumped)
-bash-4.2$ /data/scratch/tbisanz/allpix-squared/bin/allpix -c simulation_full.conf -j 1
|12:52:15.716| (INFO) Setting provided option multithreading=true
|12:52:15.717| (INFO) Setting provided option workers=1
|12:52:15.717| (STATUS) Welcome to Allpix^2 v2.0.0-1791-ge8471132b
|12:52:15.717| (STATUS) Initialized PRNG with system entropy seed 5412835038827642083
|12:52:16.780| (WARNING) Main ROOT file "/data/scratch/tbisanz/ap2_proton_therapy/output_tmp/modules.root" exists and will be overwritten.
|12:52:16.875| (STATUS) Loaded 7 modules
|12:52:16.876| (FATAL) Error in the configuration:
Value 1 of key 'workers' in global section is not valid: number of workers should be larger than one
The configuration needs to be updated. Cannot continue.
-bash-4.2$ /data/scratch/tbisanz/allpix-squared/bin/allpix -c simulation_full.conf -h
Allpix Squared
Generic Pixel Detector Simulation Framework
Usage: allpix -c <file> [OPTIONS]
Options:
-c <file> configuration file to be used
-l <file> file to log to besides standard output
-o <option> extra module configuration option(s) to pass
-g <option> extra detector configuration options(s) to pass
-v <level> verbosity level, overwriting the global level
-j <workers> number of worker threads, equivalent to
-o multithreading=true -o workers=<workers>
--version print version information and quit
For more help, please see <https://cern.ch/allpix-squared>
-bash-4.2$ /data/scratch/tbisanz/allpix-squared/bin/allpix -c simulation_full.conf -o multithreading=false
|12:52:38.978| (INFO) Setting provided option multithreading=false
|12:52:38.980| (STATUS) Welcome to Allpix^2 v2.0.0-1791-ge8471132b
|12:52:38.980| (STATUS) Initialized PRNG with system entropy seed 2232024999035314327
|12:52:40.251| (WARNING) Main ROOT file "/data/scratch/tbisanz/ap2_proton_therapy/output_tmp/modules.root" exists and will be overwritten.
|12:52:40.318| (STATUS) Loaded 7 modules
|12:52:40.318| (STATUS) Multithreading disabled
|12:52:40.318| (STATUS) Initializing 7 module instantiations
################################
!!! G4Backtrace is activated !!!
################################
|12:52:40.528| (STATUS) Initialized 7 module instantiations
|12:52:40.528| (STATUS) Starting event loop
|12:52:46.373| (STATUS) Buffered 0, finished 26 of 2000 events ^C
|12:52:46.380| (STATUS) (Event 27) [R:DepositionGeant4] Interrupted! Finishing up active events...
|12:52:46.495| (STATUS) Finished run of 27 events
|12:52:46.569| (STATUS) Finalization completed
|12:52:46.570| (WARNING) Unused configuration keys in section DetectorHistogrammer:sensor1:
max_cluster_charge
|12:52:46.570| (STATUS) Executed 7 instantiations in 6 seconds, spending 54% of time in slowest instantiation DepositionGeant4
|12:52:46.570| (STATUS) Average processing time is 221.014ms/event, event generation at 5 Hz
-bash-4.2$ /data/scratch/tbisanz/allpix-squared/bin/allpix -c simulation_full.conf -j 6
|12:52:51.948| (INFO) Setting provided option multithreading=true
|12:52:51.949| (INFO) Setting provided option workers=6
|12:52:51.949| (STATUS) Welcome to Allpix^2 v2.0.0-1791-ge8471132b
|12:52:51.949| (STATUS) Initialized PRNG with system entropy seed 17503969746766871004
|12:52:53.118| (WARNING) Main ROOT file "/data/scratch/tbisanz/ap2_proton_therapy/output_tmp/modules.root" exists and will be overwritten.
|12:52:53.189| (STATUS) Loaded 7 modules
|12:52:53.189| (STATUS) Multithreading enabled, processing events in parallel on 6 worker threads
|12:52:53.189| (STATUS) Allocating a total of 1536 event slots for buffered modules
|12:52:53.189| (STATUS) Initializing 7 module instantiations
################################
!!! G4Backtrace is activated !!!
################################
|12:52:56.518| (STATUS) Initialized 7 module instantiations
|12:52:56.523| (STATUS) Starting event loop
|12:52:58.263| (STATUS) Buffered 0, finished 29 of 2000 events ^C
|12:52:58.265| (STATUS) Interrupted! Finishing up active events...
|12:52:58.449| (STATUS) Finished run of 34 events
|12:52:58.571| (STATUS) Finalization completed
|12:52:58.572| (WARNING) Unused configuration keys in section DetectorHistogrammer:sensor1:
max_cluster_charge
|12:52:58.572| (STATUS) Executed 7 instantiations in 5 seconds, spending 56% of time in slowest instantiation ProjectionPropagation:sensor1
|12:52:58.572| (STATUS) Average processing time is 56.6458ms/event, event generation at 18 Hz
|12:52:58.572| (STATUS) This corresponds to a processing time of 339.875ms/event per worker
-bash-4.2$ /data/scratch/tbisanz/allpix-squared/bin/allpix -c simulation_full.conf
|12:53:01.498| (STATUS) Welcome to Allpix^2 v2.0.0-1791-ge8471132b
|12:53:01.499| (STATUS) Initialized PRNG with system entropy seed 4948159949323449942
|12:53:02.577| (WARNING) Main ROOT file "/data/scratch/tbisanz/ap2_proton_therapy/output_tmp/modules.root" exists and will be overwritten.
|12:53:02.663| (STATUS) Loaded 7 modules
|12:53:02.663| (STATUS) Multithreading enabled, processing events in parallel on 31 worker threads
|12:53:02.663| (STATUS) Allocating a total of 7936 event slots for buffered modules
|12:53:02.663| (STATUS) Initializing 7 module instantiations
################################
!!! G4Backtrace is activated !!!
################################
|12:53:05.793| (STATUS) Initialized 7 module instantiations
|12:53:05.830| (STATUS) Starting event loop
|12:53:20.721| (STATUS) Buffered 0, finished 372 of 2000 events ^C
|12:53:20.818| (STATUS) Interrupted! Finishing up active events...
|12:53:21.520| (STATUS) Finished run of 403 events
|12:53:21.741| (STATUS) Finalization completed
|12:53:21.742| (WARNING) Unused configuration keys in section DetectorHistogrammer:sensor1:
max_cluster_charge
|12:53:21.742| (STATUS) Executed 7 instantiations in 19 seconds, spending 68% of time in slowest instantiation ProjectionPropagation:sensor1
|12:53:21.742| (STATUS) Average processing time is 38.9338ms/event, event generation at 26 Hz
|12:53:21.742| (STATUS) This corresponds to a processing time of 1206.95ms/event per worker
Edited by Tobias Bisanz