Improve handling of file opening with xrootd in get_unprocessed_files.py
When using the Legacy
convention in the get_unprocessed_files.py
, it can happen for the xrootd client to hang forever during the file opening operation in some cases.
Unless the XRD_REQUESTTIMEOUT
env variable is set, the default xrootd timeout is 1.5h and the script can actually hang for a very long time. A timeout should be added to the script to avoid this issue. I will check how coffea enforces such behavior, but it seems that this can be solved by specifying f.open(fpath,timeout)
.
An addition would also be to provide a way to skipbadfiles
as in coffea
. This would allow for the script to complete, ignoring the access errors you can sometimes encounter while trying to read a root file.
Edited by Paul Lucas Devouge