Skip to content

GitLab

Add Swift support to FTS-REST

Open Shiting Long requested to merge swift into develop

Changes including:

  1. added authentication to Openstack Keystone for Swift. There are two ways for setting credentials (OS tokens) for Swift:
  • Manually set OS tokens through CLI
  • FTS fetch OS tokens from the Keystone server using OIDC access tokens
  1. added command-line options --os-token(OPTIONAL), --os-project-id(MANDATORY) for submitting Swift transfers.
  2. added fts_swift_token_refresh_daemon for refreshing OS tokens.
  3. added CSSwift.py for handling possible requests from WebFTS, e.g., list contents and set OS tokens for Swift.

DB change:

  1. added column os_project_id in t_job.
  2. added column keystone_url and keystone_idp in t_cloudStorage.
  3. new table t_cloudCredentialCache.

Some comments:

  1. Since t_cloudCredentialCache has no DB relation with t_cloudStorageUser, extra checks on the user are performed before setting cloud credentials. Although logically there should be a one-to-many relationship between t_cloudStorageUser and t_cloudCredentialCache, it doesn't make sense to me to add vo to t_cloudCredentialCache to form a composite foreign key with cloudStorage_name and user_dn because vo has no use for the table.
  2. Any cloud storage user would be able to indirectly alter t_cloudCredentialCache because this is linked to transfers. However, this exposes risks that users might add an arbitrary number of rows in the table, so I added an extra check before adding/setting the row to see if the added credentials are valid.
  3. Please add the required python packages (keystoneauth1 and python-keystoneclient) to the docker image so that the pipeline can work.
  4. At the time being, you can submit Swift transfer with:

fts-rest-transfer-submit -s https://<fts> --access-token $tok swifts://<source> swifts://<dest> --os-project-id "<source_project_id>:<dest_project_id>" --os-token "<project_id>:<corresponding_os_token>"

After we have FENIX AAI properly set up, you can submit the transfer without specifying --os-token.