Fix for ATLINFR-5383 : only check for files for DiffPool

Closes ATLINFR-5383

I'm really not sure how we didn't run into this before, but this adds a protection to not check for file paths if they shouldn't exist (as uncovered on !71511 (merged))

cc @abarton