fts-rest-flask merge requestshttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests2023-12-12T15:44:46+01:00https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/115Merge Alma9 changes to token project2023-12-12T15:44:46+01:00Joao Pedro LopesMerge Alma9 changes to token projectJoao Pedro LopesJoao Pedro Lopeshttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/114Build fts-rest-flask in AlmaLinux92023-12-11T17:26:29+01:00Joao Pedro LopesBuild fts-rest-flask in AlmaLinux9This merge requests introduces the necessary modifications to build AlmaLinux9 RPM'sThis merge requests introduces the necessary modifications to build AlmaLinux9 RPM'sJoao Pedro LopesJoao Pedro Lopeshttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/112FTS-1829: Adds SciTag support to the FTS REST Client and Server2023-10-13T11:30:10+02:00Mihai Patrascoiumihai.patrascoiu@cern.chFTS-1829: Adds SciTag support to the FTS REST Client and ServerThis merge request brings changes to both the client and the server for the submission of per-transfer `scitag` values.
SciTag values are validated to be in the `[65, 65535]` inclusive rage.
Client-side:
- Add the `--scitag <scitag>` ...This merge request brings changes to both the client and the server for the submission of per-transfer `scitag` values.
SciTag values are validated to be in the `[65, 65535]` inclusive rage.
Client-side:
- Add the `--scitag <scitag>` CLI option
- Add the `scitag` option to the `new_transfer(..)` function
- Validate the `scitag` value falls in the `[65, 65535]` range
- Puts the `scitag` value as a per-transfer attribute in the final submission JSON
Server-side:
- Parse the submission JSON and extract the per-transfer `scitag` value
- Validate `scitag` is an integer type with values in the `[65. 65535]` range (`None` value allowed)
- Store the transfers in the database
Functional tests have been added to test SciTag transfer submissions in the `test_job_submission.py` file.
For this branch to work, the upcoming FTS Schema 8.2.0 is needed, provided by `fts/fts3::FTS-1829_scitags` branch ([link](https://gitlab.cern.ch/fts/fts3/-/blob/FTS-1829_scitags/src/db/schema/mysql/fts-diff-8.2.0.sql)).Mihai Patrascoiumihai.patrascoiu@cern.chMihai Patrascoiumihai.patrascoiu@cern.chhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/111FTS-1927: Renamed the "fts_token" to "fts_access_token" in the client2023-10-02T11:33:07+02:00Shubhangi MisraFTS-1927: Renamed the "fts_token" to "fts_access_token" in the clientRenamed the "fts_token" to "fts_access_token" in the clientRenamed the "fts_token" to "fts_access_token" in the clientSteven MurraySteven Murrayhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/106FTS-1950 : Renamed the source and destination space token options in FTS2023-10-02T10:47:18+02:00Shubhangi MisraFTS-1950 : Renamed the source and destination space token options in FTSFTS-1950 : Renamed the source and destination space token options in FTS transfer submission for CLI and rest server
Closes FTS-1950FTS-1950 : Renamed the source and destination space token options in FTS transfer submission for CLI and rest server
Closes FTS-1950https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/107Resolve FTS-1962 by having the Submitter:build_submission() method delete...2023-09-18T15:56:41+02:00Steven MurrayResolve FTS-1962 by having the Submitter:build_submission() method delete...Resolve FTS-1962 by having the Submitter:build_submission() method delete job["params"]["archive_metadata"]
Closes FTS-1962Resolve FTS-1962 by having the Submitter:build_submission() method delete job["params"]["archive_metadata"]
Closes FTS-1962https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/104FTS-1932: FTS CLI transfer access tokens for source and destination2023-08-16T17:17:22+02:00Shubhangi MisraFTS-1932: FTS CLI transfer access tokens for source and destinationThis merge request modifies the FTS CLI such that it can accept access tokens for the source and the destination.
Closes FTS-1932This merge request modifies the FTS CLI such that it can accept access tokens for the source and the destination.
Closes FTS-1932https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/105FTS-1947: Provide CLI access token compatibility2023-08-16T17:03:33+02:00Shubhangi MisraFTS-1947: Provide CLI access token compatibilityThis merge request modifies the FTS CLI in order to provide backwards compatibility for the `--access-token` option.
In the future, this option will be removed.
Should be merged into `FTS-1932_cli_transfer_access_token` branch, before...This merge request modifies the FTS CLI in order to provide backwards compatibility for the `--access-token` option.
In the future, this option will be removed.
Should be merged into `FTS-1932_cli_transfer_access_token` branch, before https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/104
Closes FTS-1947https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/101FTS-1923: Turn configurable eviction flag into skip-eviction2023-08-03T17:39:40+02:00Joao Pedro LopesFTS-1923: Turn configurable eviction flag into skip-evictionWIP: Needs https://gitlab.cern.ch/fts/fts3/-/merge_requests/53 to be merged first
FTS-1923: Turn configurable eviction flag into skip-eviction in order to not perform file release for Storage ElementWIP: Needs https://gitlab.cern.ch/fts/fts3/-/merge_requests/53 to be merged first
FTS-1923: Turn configurable eviction flag into skip-eviction in order to not perform file release for Storage ElementMihai Patrascoiumihai.patrascoiu@cern.chMihai Patrascoiumihai.patrascoiu@cern.chhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/103FTS-1927: Removed access-token option from CLI and added fts-token option2023-08-02T15:25:21+02:00Shubhangi MisraFTS-1927: Removed access-token option from CLI and added fts-token optionCloses FTS-1927Closes FTS-1927https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/96FTS-1885: Refuse FTS job submissions if archive/staging metadata exceeds conf...2023-08-01T15:19:50+02:00Shubhangi MisraFTS-1885: Refuse FTS job submissions if archive/staging metadata exceeds configurable sizeThe `staging-metadata` and `archive-metadata` fields are JSON objects. More so, the experiment frameworks store large amounts of metadata per file. To prevent storing very large metadata fields in FTS, a size restriction is needed (confi...The `staging-metadata` and `archive-metadata` fields are JSON objects. More so, the experiment frameworks store large amounts of metadata per file. To prevent storing very large metadata fields in FTS, a size restriction is needed (configurable, default 1kb).
If the metadata field exceeds the limit, the REST Server will return an error message upon job submission.
The staging/archive metadata for each file size has to be checked independently.Shubhangi MisraShubhangi Misrahttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/100FTS-1920: Adapt schema for the transfer retry log file change2023-07-24T17:09:41+02:00Mihai Patrascoiumihai.patrascoiu@cern.chFTS-1920: Adapt schema for the transfer retry log file changeAdapts `FTS-REST-Flask` to the new schema brought by FTS-1920.
Needs https://gitlab.cern.ch/fts/fts3/-/merge_requests/50Adapts `FTS-REST-Flask` to the new schema brought by FTS-1920.
Needs https://gitlab.cern.ch/fts/fts3/-/merge_requests/50https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/98FTS-1914: Extend the config/se endpoint to allow the configuration of the...2023-07-20T14:18:42+02:00Joao Pedro LopesFTS-1914: Extend the config/se endpoint to allow the configuration of the...FTS-1914: Extends the config/se endpoint to allow the configuration of the supported TPC modes per Storage EndpointFTS-1914: Extends the config/se endpoint to allow the configuration of the supported TPC modes per Storage EndpointMihai Patrascoiumihai.patrascoiu@cern.chMihai Patrascoiumihai.patrascoiu@cern.chhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/97FTS-1763: Disallow submission of deletion jobs2023-04-14T11:58:22+02:00Joao Pedro LopesFTS-1763: Disallow submission of deletion jobsFTS-1763: Disallow submission of deletion jobs by return 405 'Method Not Allowed' status code on such submissions
Closes FTS-1763FTS-1763: Disallow submission of deletion jobs by return 405 'Method Not Allowed' status code on such submissions
Closes FTS-1763Joao Pedro LopesJoao Pedro Lopeshttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/92Do not crash if the authorityKeyIdentifier is not in the certificate (OpenSSLv3)2023-03-03T10:59:59+01:00Christophe HaenDo not crash if the authorityKeyIdentifier is not in the certificate (OpenSSLv3)When running with OpenSSL3, we noticed the following exception upon delegation, that can be reproduced easily:
```python
python -c "import fts3.rest.client.easy as fts3;ctx = fts3.Context(endpoint='https://fts3-lhcb.cern.ch:8446', ucert...When running with OpenSSL3, we noticed the following exception upon delegation, that can be reproduced easily:
```python
python -c "import fts3.rest.client.easy as fts3;ctx = fts3.Context(endpoint='https://fts3-lhcb.cern.ch:8446', ucert='/tmp/x509up_u56212',verify=False);fts3.delegate(ctx,force=True)"
Traceback (most recent call last):
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 282, in delegate
x509_proxy = self._sign_request(x509_request, lifetime)
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 187, in _sign_request
identifier_ext = _workaround_new_extension(
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 71, in _workaround_new_extension
x509_ext_ptr = m2.x509v3_ext_conf(None, ctx, name, value)
M2Crypto.X509.X509Error: unable to get issuer keyid
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/easy/delegate.py", line 40, in delegate
return delegator.delegate(lifetime, force, delegate_when_lifetime_lt)
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 291, in delegate
raise ClientError(str(e)).with_traceback(sys.exc_info()[2])
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 282, in delegate
x509_proxy = self._sign_request(x509_request, lifetime)
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 187, in _sign_request
identifier_ext = _workaround_new_extension(
File "/opt/dirac/versions/v8.1.0a7-1676534507/Linux-x86_64/lib/python3.9/site-packages/fts3/rest/client/delegator.py", line 71, in _workaround_new_extension
x509_ext_ptr = m2.x509v3_ext_conf(None, ctx, name, value)
fts3.rest.client.exceptions.ClientError: Client error: unable to get issuer keyid
```
The problem is due to proxies being a grey area when it comes to Authority Key Identifier (https://www.rfc-editor.org/rfc/rfc3280#section-4.2.1.1). In any case, this being a non critical extension, the sensible thing to do is probably to just skip it.
A similar PR was made against VOMS: https://github.com/italiangrid/voms/pull/113https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/94FTS-1887 Log how long in takes to store an FTS job into the database2023-02-28T09:53:02+01:00Steven MurrayFTS-1887 Log how long in takes to store an FTS job into the databaseCloses FTS-1887Closes FTS-1887https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/91FTS-1857: Introduce "archive_metadata" file property2022-11-17T11:28:14+01:00Joao Pedro LopesFTS-1857: Introduce "archive_metadata" file propertyThis commit introduces the following changes:
- Introduces "--archive-metadata" as a new parameter for file transfers"
- Stores the submitted "archive_metadata" per file in the "t_file" table
- Sends the "archive_metadata" in the "...This commit introduces the following changes:
- Introduces "--archive-metadata" as a new parameter for file transfers"
- Stores the submitted "archive_metadata" per file in the "t_file" table
- Sends the "archive_metadata" in the "state_change" monit messages
Closes FTS-1857https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/90Profile jobs2022-07-18T17:18:49+02:00Tom HepworthProfile jobsCreate new decorator for `/jobs` endpoints to get a better understanding of how users use the service by returning logs specifying user VO, request type, response status. This will eventually be picked up by fluent-bit, parsed, and publi...Create new decorator for `/jobs` endpoints to get a better understanding of how users use the service by returning logs specifying user VO, request type, response status. This will eventually be picked up by fluent-bit, parsed, and published to graphite, to be shown on a grafana dashboard.Tom HepworthTom Hepworthhttps://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/89FTS-1709: Add 'os_project_id' field to model/job.py2022-07-08T10:58:59+02:00Shiting LongFTS-1709: Add 'os_project_id' field to model/job.pycan safely delete this after merge.can safely delete this after merge.https://gitlab.cern.ch/fts/fts-rest-flask/-/merge_requests/88FTS-1807: Introduce staging_metadata2022-07-08T10:35:36+02:00Joao Pedro LopesFTS-1807: Introduce staging_metadataThis merge request introduces the concept of `staging_metadata` in the `fts-rest-flask` project.
1. It introduces `--staging-metadata` as a new parameter for file transfer submissions
2. Stores the given `staging_metadata` as a file pro...This merge request introduces the concept of `staging_metadata` in the `fts-rest-flask` project.
1. It introduces `--staging-metadata` as a new parameter for file transfer submissions
2. Stores the given `staging_metadata` as a file property
3. Sends the given `staging_metadata` in the monitoring messages