Lower limit on amount of run_athena jobs that should be run
As shown in this pipeline we can completely miss cases where log.generate
files are not provided.
Frank suggested the following check that can be implemented in e.g. the run_athena
job:
if Nlogfiles < roundup(0.01*N_newDSIDs)
return 1