fts-rest-flask merge requestshttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests2024-03-06T10:13:57+01:00https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/116Update easy interface with source_token and destination_token2024-03-06T10:13:57+01:00Christophe HaenUpdate easy interface with source_token and destination_tokenI'm currently trying to see how I could get tokens to work in FTS for LHCb, so I'll keep here the changes I make to the clientI'm currently trying to see how I could get tokens to work in FTS for LHCb, so I'll keep here the changes I make to the clienthttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/87client: add support for metalink source2024-02-24T22:11:22+01:00Paul Millarclient: add support for metalink sourceMotivation:
FTS supports bulk operations, but using its own specific format. The
metalink is a well-documented (see RFC 5854) XML format that describes
how to receive a set of files.
Support for metalink would allow FTS to accept a bu...Motivation:
FTS supports bulk operations, but using its own specific format. The
metalink is a well-documented (see RFC 5854) XML format that describes
how to receive a set of files.
Support for metalink would allow FTS to accept a bulk transfer request
with the FTS user presenting the information in a standard fashion.
Modification:
Add initial support for parsing the metalink XML format. The idea is
that the user presents a base URL that all files (within the metalink
file) are resolved against.
There are several limitations with the current implementation:
* Metalink allows for a file to have multiple sources. This patch
selects only the best url (the one with the lowest priority value).
* There are several types of source URLs supported by metalink (http,
bittorrent, etc.). The code currently assumes that any source URL
is acceptable by FTS
* In principle, the metalink allows a file to have different digest
values, as calculated by different checksum algorithms. Currently,
only adler32 checksums are supported.
Result:
It is now possible to submit a bulk request to FTS using a metalink
file.https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/113FTS-1965: Validate activity shares configuration2023-10-23T11:25:09+02:00Joao Pedro LopesFTS-1965: Validate activity shares configurationCurrently the FTS-REST interface stores the activity shares configuration in the DB in plain text format. Later the FTS server reads that text and expects it to be equivalent to a JSON document. Therefore, the text stored in the DB must ...Currently the FTS-REST interface stores the activity shares configuration in the DB in plain text format. Later the FTS server reads that text and expects it to be equivalent to a JSON document. Therefore, the text stored in the DB must be equivalent to following JSON document: `[{"A": 1}, {"B": 2}]`, otherwise the server won't be able to read it.
On the REST-API the full format of the submission should be one of the following:
```json
{
"vo": "dteam",
"share": [
{
"A": 1
},
{
"B": 2
}
],
"active": true
}
```
```json
{
"vo": "dteam",
"share": {
"A": 1,
"B": 2
},
"active": true
}
```
In the first one the "share" object is already in the expected format. On the second one the REST-API will convert the "share" dictionary into the appropriate format to be read by the FTS server.
This merge request tries to make sure that the REST interface will only write the activities shares in the DB in the right format expected by the server.
In the future the whole mechanism of configuring activities should be revisited. And the DB should be modeled to store the activity shares configuration rather that storing a whole JSON document.
Closes FTS-1965Joao Pedro LopesJoao Pedro Lopeshttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/110FTS-1964: Return HTTP 201 status code when creating a new link configuration2023-09-26T13:56:19+02:00Joao Pedro LopesFTS-1964: Return HTTP 201 status code when creating a new link configurationThis merge request modifies the FTS-REST interface to return the HTTP 201 status code when a new link configuration is created.
Closes FTS-1964This merge request modifies the FTS-REST interface to return the HTTP 201 status code when a new link configuration is created.
Closes FTS-1964Mihai Patrascoiumihai.patrascoiu@cern.chMihai Patrascoiumihai.patrascoiu@cern.chhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/109Resolve FTS-1961 "Se config"2023-09-26T13:46:45+02:00Joao Pedro LopesResolve FTS-1961 "Se config"This merge request fixes the bug described in FTS-1961:
> When trying to configure a Storage Endpoint (SE) via the FTS REST interface https://<fts-instance>:8446/config/se if the provided JSON object does not contain the `se_info` value...This merge request fixes the bug described in FTS-1961:
> When trying to configure a Storage Endpoint (SE) via the FTS REST interface https://<fts-instance>:8446/config/se if the provided JSON object does not contain the `se_info` value or if the `se_info` value is an empty JSON an error is thrown on the server side.
It also adds extra testing to validate the different submission scenarios for SE configuration.Mihai Patrascoiumihai.patrascoiu@cern.chMihai Patrascoiumihai.patrascoiu@cern.chhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/108FTS-1967: Prevent invalid values of tpc_support to be stored in the t_se table2023-09-26T12:02:25+02:00Joao Pedro LopesFTS-1967: Prevent invalid values of tpc_support to be stored in the t_se tableThis merge request fixes a bug that was introduced when https://its.cern.ch/jira/browse/FTS-1914 was implemented.
The REST interface inserted the word "test" on the `t_se` table if a configuration already existed for a given SE and the ...This merge request fixes a bug that was introduced when https://its.cern.ch/jira/browse/FTS-1914 was implemented.
The REST interface inserted the word "test" on the `t_se` table if a configuration already existed for a given SE and the button "save" was pressed. This bug is fixed here and in addition a modification is made to never allow the submission of a "tpc_support" value which is not valid.
A test case to test the submission being refused if the "tpc_support" was invalid was also added.Mihai Patrascoiumihai.patrascoiu@cern.chMihai Patrascoiumihai.patrascoiu@cern.chhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/85Add Swift support to FTS-REST2022-05-06T15:52:32+02:00Shiting LongAdd Swift support to FTS-RESTChanges including:
1. added authentication to Openstack Keystone for Swift. There are two ways for setting credentials (OS tokens) for Swift:
- Manually set OS tokens through CLI
- FTS fetch OS tokens from the Keystone server using OIDC ...Changes including:
1. added authentication to Openstack Keystone for Swift. There are two ways for setting credentials (OS tokens) for Swift:
- Manually set OS tokens through CLI
- FTS fetch OS tokens from the Keystone server using OIDC access tokens
2. added command-line options `--os-token`(OPTIONAL), `--os-project-id`(MANDATORY) for submitting Swift transfers.
3. added `fts_swift_token_refresh_daemon` for refreshing OS tokens.
4. added `CSSwift.py` for handling possible requests from WebFTS, e.g., list contents and set OS tokens for Swift.
DB change:
1. added column `os_project_id` in `t_job`.
2. added column `keystone_url` and `keystone_idp` in `t_cloudStorage`.
3. new table `t_cloudCredentialCache`.
Some comments:
1. Since `t_cloudCredentialCache` has no DB relation with `t_cloudStorageUser`, extra checks on the user are performed before setting cloud credentials. Although logically there should be a one-to-many relationship between `t_cloudStorageUser` and `t_cloudCredentialCache`, it doesn't make sense to me to add `vo` to `t_cloudCredentialCache` to form a composite foreign key with `cloudStorage_name` and `user_dn` because `vo` has no use for the table.
2. Any cloud storage user would be able to indirectly alter `t_cloudCredentialCache` because this is linked to transfers. However, this exposes risks that users might add an arbitrary number of rows in the table, so I added an extra check before adding/setting the row to see if the added credentials are valid.
3. Please add the required python packages (keystoneauth1 and python-keystoneclient) to the docker image so that the pipeline can work.
4. At the time being, you can submit Swift transfer with:
> fts-rest-transfer-submit -s https://\<fts> --access-token $tok swifts://\<source> swifts://\<dest> --os-project-id "<source_project_id>:<dest_project_id>" --os-token "<project_id>:<corresponding_os_token>"
After we have FENIX AAI properly set up, you can submit the transfer without specifying `--os-token`.