Lower limit on amount of run_athena jobs that should be run
As shown in this pipeline we can completely miss cases where log.generate files are not provided.
Frank suggested the following check that can be implemented in e.g. the run_athena job:
if Nlogfiles < roundup(0.01*N_newDSIDs)
return 1