Pythia reproducibility issue

Investigating the non-reproducibility of Pythia8 minimum bias events in the multi-threaded framework. The executable compareHepMCEvents.exe can compare two HepMC3 files. The problem has been narrowed down to the case where the order of events (i.e. the order of seeds given to pythia) can be different. This occurs no matter on whether multiple Pythia8 instances are used concurrently or only one locking instance is used.

Augmenting the random engine wrapper to dump all generated random numbers and the call backtrace into a file revealed that the actual sequence of random numbers is always identical. However, the backtraces reveal that the calling functions suddenly divergence as can be seen when comparing these two files:

run1.txt run2.txt

Based on that I suspect it happens somehow in PhaseSpace2to2diffractive::trialKin where the stopping criteria for the internal loop behaves differently due to different values for the state of the pythia instance.

Another example occurred concerning the showering where I have the two corresponding logs attached:

run2B.txt run1B.txt

I also found cases where the produced HepMC events are slightly different (but clearly not just due to floating point precision) but the produced random number sequences and the back-traces where I again suspect some internal state being.