NLHE check not working well in CI
As seen in !1304 (merged).
Maybe this check could be based on the extrapolated number of events?
For a sufficiently large
nEventsPerJobthe 10% is more conservative than 4 sigma, so no harm done, albeit a bit more inefficient. For low
nEventsPerJob4sigma is more conservative and people would probably anyway notice this in their local runs and choose a larger margin (e.g. factors of 25 rather than 1.1). So arguably the 4sigma thing is probably not needed in practice, provided people actually test their setups locally...
Maybe @jkretz should indicate what his preference is for the long run, but at the moment this new rule is causing the CI to crash frequently for no good reason, so we need to patch somehow.
Another option is to remove the 4sigma rule and go back to the previous check with 10% extra events, or that we have the 4sigma rule only for jobs that have a small number of requested output events (although it's actually impossible to know if people just test with a low number of events)