[apd.snakemake] using modification time provided by LbAPI instead of using XRootD stat as it takes too long for O(1000) PFNs
when XRootD evaluates the DAG it runs stat
to find the last modification time of each of the PFNs returned by apd
, which can take an extremely long time (possibly memory issues too) when trying to run workflows with large datasets
I was hoping that we could return a modified form of the XRootDRemoteProvider that patches the exists
methods for the remote objects and returns the last modified time from the LbAPI response instead of running stat
on each PFN. This way the DAG evaluation becomes very fast, using the data already possessed by apd
.
Is this possible to implement without too much effort?