DatabaseWriter: Database Connections per Thread, Transactions, Cleanup
- make individual database connections per thread as suggested by https://libpqxx.readthedocs.io/en/7.3.0/a01348.html
- implement
waive_sequence_requirement
to wwirte directly instead of single-threaded buffered. This gives a massive performance boost:
With waive_sequence_requirement = false
|21:31:23.099| (STATUS) Welcome to Allpix^2 v2.0.0+246^g4b0a00fc
|21:31:23.099| (STATUS) Initialized PRNG with configured seed 0
|21:31:23.099| (STATUS) Initialized core PRNG with configured seed 0
|21:31:23.386| (STATUS) Loaded 7 modules
|21:31:26.092| (STATUS) Initialized 27 module instantiations
|21:31:26.092| (STATUS) Multithreading enabled, processing events in parallel on 7 worker threads
|21:31:26.092| (STATUS) Allocating a total of 3584 event slots for buffered modules
|21:31:26.097| (STATUS) Starting event loop
|21:32:33.964| (STATUS) Finished run of 1000 events
|21:32:33.980| (STATUS) [F:DatabaseWriter] Wrote 52945 objects from 18998 messages to database
|21:32:33.983| (STATUS) Finalization completed
|21:32:33.983| (STATUS) Executed 27 instantiations in 1 minutes 11 seconds, spending 88% of time in slowest instantiation DatabaseWriter
|21:32:33.983| (STATUS) Average processing time is 71 ms/event, event generation at 14 Hz
|21:32:33.983| (STATUS) This corresponds to a processing time of 497 ms/event per worker
With waive_sequence_requirement = true
|21:31:06.991| (STATUS) Welcome to Allpix^2 v2.0.0+246^g4b0a00fc
|21:31:06.991| (STATUS) Initialized PRNG with configured seed 0
|21:31:06.991| (STATUS) Initialized core PRNG with configured seed 0
|21:31:07.183| (STATUS) Loaded 7 modules
|21:31:09.874| (STATUS) Initialized 27 module instantiations
|21:31:09.874| (STATUS) Multithreading enabled, processing events in parallel on 7 worker threads
|21:31:09.874| (STATUS) Allocating a total of 3584 event slots for buffered modules
|21:31:09.879| (STATUS) Starting event loop
|21:31:19.961| (STATUS) Finished run of 1000 events
|21:31:19.976| (STATUS) [F:DatabaseWriter] Wrote 52945 objects from 18998 messages to database
|21:31:19.980| (STATUS) Finalization completed
|21:31:19.980| (STATUS) Executed 27 instantiations in 13 seconds, spending 86% of time in slowest instantiation DatabaseWriter
|21:31:19.981| (STATUS) Average processing time is 13 ms/event, event generation at 78 Hz
|21:31:19.981| (STATUS) This corresponds to a processing time of 91 ms/event per worker
This fixes #209 (closed)
(cc) @schioppa
Edited by Simon Spannagel