Skip to content

Cache task list to speed up submit.py

Moritz Hesping requested to merge cache-task-list into master

When running over a large number of files, submit.py gets painfully slow. In particular, even with --checkmissing it can take a long time. The reason for this is that the task list is recalculated every time. I have added some code that dumps the task list to a file via pickle. In future calls of submit.py (with the same identifier) it is then loaded from the file (if it is still up to date, i.e. younger than the job list), thereby saving a lot of time.

Merge request reports

Loading