Severe performance degradation when using HEPMC generator and tracking plugin, and cannot process files with more than 10 events.
When using an input HEPMC file with more than just a few vertices in it and the tracks plugin fullSimLight/FSL effectively hangs. I have had success running a small example file (7 vertices 12 particles), but when running a slightly larger file (31 vertices, 42 particles) I experience the problem. I have noticed that when running larger files, if I watch the size of the output .h5 file it will rapidly grow, roughly 40mb per second on my hard drive, until it reaches over ~600mb, at which point the the write speed drops to ~0.1mb per second. Interestingly, I have also noticed that if I turn on the tracking verbosity (any level 1-5), nothing writes to the *.h5 file, but if I pipe the terminal output to a file, this file grows at a rate of around 40mb per second, and I can see in bpytop that my CPU is working much harder. My guess then is that this has something to do with the hdf5 file writing process making some sort of bottleneck as the files are quite large...
Finally, I have tried producing a HEPMC that has each vertex placed in it's own event. In my example there are 78 vertices/events, when I run this file fullSimLight quits after processing 10 of the events/vertices and gives this output (sorry for the formatting I'm not sure how to get it to just display plain text):
-----------------------------------------------------------------------------------------------------------------------------------------
GenEvent: #9
Momentum units: GEV Position units: MM
Entries in this event: 1 vertices, 2 particles, 0 weights.
Position offset: 0, 0, 0, 0
GenParticle Legend
ID PDG ID ( px, py, pz, E ) Stat ProdVtx
________________________________________________________________________
Vtx: -1 stat: 0 (X,cT): 1.7e-07 -4.3e-07 2.7e-05 2.7e-05
I: 1 -311 +7.89e-02,-1.97e-01,+1.24e+01,+1.24e+01 2 0
O: 2 130 +7.89e-02,-1.97e-01,+1.24e+01,+1.24e+01 1 -1
________________________________________________________________________
--------------------------------
--------- EVENT CONTENT --------
--------------------------------
Weights (0):
Attributes:
GenParticlePtr (2)
GenParticle: 1 PDGID: -311 (P,E)=+7.89e-02,-1.97e-01,+1.24e+01,+1.24e+01 Stat: 2 PV: 0 EV: -1 Attr: 0
GenParticle: 2 PDGID: 130 (P,E)=+7.89e-02,-1.97e-01,+1.24e+01,+1.24e+01 Stat: 1 PV: -1 EV: 0 Attr: 0
GenVertexPtr (1)
GenVertex: -1 stat: 0 in: 1 out: 1 has_set_position: true (X,cT): 1.7135e-07, -4.28113e-07, 2.69565e-05, 2.69821e-05
-----------------------------
G4Worker_0 > Particle pdgcode: 130
G4Worker_0 > HepMC3PrimaryGeneratorAction::GeneratePrimaries MUTEX unlocked
The file Tracks_data_23.h5 has been created.
=================================================================
Final random number = 0.330781
=================================================================
double free or corruption (fasttop)
### CAUGHT SIGNAL: 6 ### address: 0x3e80000822f, signal = SIGABRT, value = 6, description = abort program (formerly SIGIOT).
malloc_consolidate(): unaligned fastbin chunk detected
Aborted (core dumped)
----------------------------------------------------------------------------------------------------------------------------------------
The output .h5 file here has a size of 290mb. I have noticed in the terminal output that after reading the hepmc input file FSL/fullSimLight outputs a formatted table of vertices using the HepMC.read() function however it shows event#0 with 78 vertices, so I guess it takes the 78 vertices and puts them in 1 event, but if I do this myself, I get the stalling behavior I described above.
An aside, I also noticed in FSL if the HEPMC generator is selected multithreading is disabled, or atleast you can't choose more than 1 thread there. I can alter the config file manually and run it with fullSimLight specifying more threads, but can easily freeze my system if I choose 4 cores as it maxes out my system memory (16gb). I wondered if maybe these things are related but I don't experience the problem when using Pythia as the generator.