Race conditions in checkloop

There are three files that are processed by both tasks: The datasets_file that tracks the progress of processing of all datasets, together with the sample_status files that track the status of individual samples' processing, and the print_status_file that contains the information from running pbook.

The latter, however, is only written in the UpdateDatasetStatus task and read in CheckSample. So, the worst that can happen is that the CheckSample tasks may run with outdated information. I think this is fine, still we don't require the progress to be tracked with high accuracy. We only care for the processing to be completed, in which case it would be definitely right, because static.

For the datasets_file and the sample_status files, it is a bit more convoluted. Multiple samples make up a dataset in this tool. So in CheckSample the dataset file is read in order to get the information about which samples are in there and read out in the run context static information like the related taskID or output name, that is required as an input to the grid tools for getting their processing's progress and status. This information is then used to write the sample specific sample_status files. In the second loop in UpdateDatasetStatus the static structure of the datasets_file is used to loop through all the samples, and for each sample the sample_status file is read to get the current status and progress. This information is then gathered and finally moved into a new temporary instance of the datasets_file which then is replacing the old one. So what can happen is that the dynamic dynamic information in there, status and progress, can be off. But they are never lost, since only written by one of the tasks. Also this information is only used to check whether something in the submission was failing (status), before the loops, which would trigger a retry anyway, and to check in the processloop whether a sample is ready to be processed (progress > 0), which if mistakenly not yet updated will be checked again in a later iteration anyway.

So, I think it is not a real issue, even though there are indeed race conditions. Nonetheless, it is better of course to avoid them altogether.