How to access data for testing?
In #13 (closed) @ngrieser is running into the issue that he can't access data for the test script as it's not on disk.
I thought we'd be okay using cern-swtest and passing -o /LocalSite/InputDataDirectory=/eos/lhcb/cern-swtest/lhcb/LHCb/Collision16/RDST/00051630/0000
. Indeed this works for my user account but not @ngrieser, the relavent part of the verbose log is:
Click to expand
2023-09-15 11:22:23 UTC dirac-production-request-run-local/InputDataResolution [139795103635264] INFO: Job has a specific policy setting: DIRAC.WorkloadManagementSystem.Client.DownloadInputData
2023-09-15 11:22:23 UTC dirac-production-request-run-local/InputDataResolution [139795103635264] INFO: Attempting to run DIRAC.WorkloadManagementSystem.Client.DownloadInputData
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.DataManagementSystem.Utilities.DMSHelpers [139795103635264] DEBUG: Grid Types are: LCG, DIRAC, VAC, CLOUD
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] INFO: InputData to be downloaded is:
/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] VERBOSE: Replicas to download are:
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] VERBOSE: /lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] INFO: Failed to find data at local SEs, will try to download from anywhere /lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] VERBOSE: GUID 584B6837-CA28-E611-8AD5-003048F0E5A4
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] VERBOSE: Size 1329219068
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] INFO: Total size of files to be downloaded is 1329219068 bytes
2023-09-15 11:22:23 UTC dirac-production-request-run-local/Subprocess [139795103635264] VERBOSE: shellCall: df -P -m /tmp/ngrieser/input-data | tail -1
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] VERBOSE: Enough disk space available (141848215552 bytes)
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] INFO: Trying to download from any SE
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] VERBOSE: Attempting to download file from all SEs (RAL-RDST): /lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/getStatus [139795103635264] DEBUG: determining status of RAL-RDST.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__executeMethod [139795103635264] DEBUG: preparing the execution of getFileMetadata
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__executeMethod [139795103635264] DEBUG: Attempting to perform 'getFileMetadata' operation with 1 lfns.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/isValid [139795103635264] DEBUG: Determining if the StorageElement RAL-RDST is valid for VO lhcb
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/isValid [139795103635264] DEBUG: Determining if the StorageElement RAL-RDST is valid for operation 'getFileMetadata'
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/getStatus [139795103635264] DEBUG: determining status of RAL-RDST.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/isValid [139795103635264] DEBUG: check the operation: CheckAccess
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__filterPlugins [139795103635264] DEBUG: Filtering plugins for getFileMetadata (protocol = None ; inputProtocol = None)
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__filterPlugins [139795103635264] DEBUG: Allowed protocol: ['root', 'https']
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__filterPlugins [139795103635264] DEBUG: Potential protocols ['root', 'https']
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/LocalSE [139795103635264] DEBUG: Determining whether RAL-RDST is a local SE.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.DataManagementSystem.Utilities.DMSHelpers [139795103635264] DEBUG: Grid Types are: LCG, DIRAC, VAC, CLOUD
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__filterPlugins [139795103635264] DEBUG: Plugins to be used for getFileMetadata: ['CTA', 'GFAL2_HTTPS']
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__executeMethod [139795103635264] DEBUG: Generating 1 protocol URLs for CTA.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__generateURLDict [139795103635264] DEBUG: generating url dict for 1 lfn in RAL-RDST.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__executeMethod [139795103635264] DEBUG: Attempting to perform 'getFileMetadata' for 1 physical files
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.CTAStorage/RAL-ANTARES [139795103635264] DEBUG: GFAL2_StorageBase.getFileMetadata: trying to read metadata for 1 paths
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.CTAStorage/RAL-ANTARES [139795103635264] DEBUG: GFAL2_StorageBase._getSingleFileMetadata: trying to read metadata for root://x509up_u85461@antares.stfc.ac.uk//eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.CTAStorage/RAL-ANTARES/GFAL2_StorageBase._getSingleMetadata [139795103635264] DEBUG: Reading metadata for root://x509up_u85461@antares.stfc.ac.uk//eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST] [139795103635264] VERBOSE: Failure in plugin to perform getFileMetadata Plugin: CTA lfn: /lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst error GError('[gfal2_stat][gfal_plugin_statG][gfal_xrootd_statG] Failed to stat file (Invalid exchange)', 52)
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__executeMethod [139795103635264] DEBUG: Generating 1 protocol URLs for GFAL2_HTTPS.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__generateURLDict [139795103635264] DEBUG: generating url dict for 1 lfn in RAL-RDST.
2023-09-15 11:22:23 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST]/__executeMethod [139795103635264] DEBUG: Attempting to perform 'getFileMetadata' for 1 physical files
2023-09-15 11:22:23 UTC dirac-production-request-run-local/GFAL2_HTTPSStorage [139795103635264] DEBUG: GFAL2_StorageBase.getFileMetadata: trying to read metadata for 1 paths
2023-09-15 11:22:23 UTC dirac-production-request-run-local/GFAL2_HTTPSStorage [139795103635264] DEBUG: GFAL2_StorageBase._getSingleFileMetadata: trying to read metadata for https://antares.stfc.ac.uk:9000/eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:23 UTC dirac-production-request-run-local/GFAL2_HTTPSStorage/GFAL2_StorageBase._getSingleMetadata [139795103635264] DEBUG: Reading metadata for https://antares.stfc.ac.uk:9000/eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.Resources.Storage.StorageElement/SE[RAL-RDST] [139795103635264] VERBOSE: Failure in plugin to perform getFileMetadata Plugin: GFAL2_HTTPS lfn: /lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst error GError('[gfal2_stat][gfal_plugin_statG][gfal_http_stat] Result HTTP 404 : File not found after 1 attempts', 2)
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.Core.Tornado.Client.ClientSelector [139795103635264] DEBUG: Trying to autodetect client for Accounting/DataStore
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.Core.Tornado.Client.ClientSelector [139795103635264] DEBUG: URL resolved: dips://lbvobox306.cern.ch:9166/Accounting/DataStoreHelper,dips://lbvobox308.cern.ch:9166/Accounting/DataStoreHelper,dips://lbvobox316.cern.ch:9166/Accounting/DataStoreHelper,dips://lbvobox311.cern.ch:9166/Accounting/DataStoreHelper
2023-09-15 11:22:24 UTC dirac-production-request-run-local [139795103635264] DEBUG: Discovering URL for service Accounting/DataStore -> dips://lbvobox311.cern.ch:9166/Accounting/DataStoreHelper
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.MonitoringSystem.Client.DataOperationSender [139795103635264] DEBUG: Concluding the sending and committing data operation to accounting
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.MonitoringSystem.Client.DataOperationSender [139795103635264] DEBUG: Committing to accounting concluded
2023-09-15 11:22:24 UTC dirac-production-request-run-local [139795103635264] DEBUG: Fail to create Producer: Requested destination not found in the CS: Monitoring::Queues::dirac.monitoring
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.MonitoringSystem.Client.DataOperationSender [139795103635264] DEBUG: Committing data operation to monitoring
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DIRAC.MonitoringSystem.Client.DataOperationSender [139795103635264] DEBUG: Committing to monitoring concluded
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] ERROR: Download from best SE failed Tried downloading /lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst: Unable to download the file from any SE
2023-09-15 11:22:24 UTC dirac-production-request-run-local/DownloadInputData [139795103635264] WARN: The following LFN(s) could not be downloaded to the WN:
/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:24 UTC dirac-production-request-run-local/InputDataResolution [139795103635264] INFO: DIRAC.WorkloadManagementSystem.Client.DownloadInputData failed for the following files:
/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:24 UTC dirac-production-request-run-local/InputDataResolution [139795103635264] ERROR: Failed to obtain access to the following files:
/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
2023-09-15 11:22:24 UTC dirac-production-request-run-local/API [139795103635264] WARN: Input data resolution failed
The issue is that he can't even access files at RAL
For me I see:
$ gfal-ls https://antares.stfc.ac.uk:9000/eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
https://antares.stfc.ac.uk:9000/eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
Whereas @ngrieser sees:
$ gfal-ls https://antares.stfc.ac.uk:9000/eos/antares/prod/lhcb/LHCb/Collision16/RDST/00051630/0000/00051630_00001081_1.rdst
gfal-ls error: 2 (No such file or directory) - Result HTTP 404 : File not found after 1 attempts
Maybe we should have a procedure for staging some files for tests rather than relying on fragile hacks.