Cache task list to speed up submit.py
When running over a large number of files, submit.py
gets painfully slow. In particular, even with --checkmissing
it can take a long time. The reason for this is that the task list is recalculated every time. I have added some code that dumps the task list to a file via pickle
. In future calls of submit.py
(with the same identifier) it is then loaded from the file (if it is still up to date, i.e. younger than the job list), thereby saving a lot of time.