Commit 4db0880c authored by Andrea Manzi's avatar Andrea Manzi
Browse files

Merge branch 'develop'

parents 44e5e4b2 89fdc2d3
Pipeline #968475 passed with stage
in 2 minutes and 25 seconds
......@@ -96,6 +96,9 @@ Usage: fts-rest-delegate [options]
-f/--force
: Force the delegation
-H/--hours
: Duration of the delegation in hours (default: 12)
### Example
```
......@@ -103,12 +106,11 @@ $ fts-rest-delegate -s https://fts3-devel.cern.ch:8446
Delegation id: 9a4257f435fa2010"
```
## fts-rest-snapshot
This command can be used to retrieve the internal status FTS3 has on all pairs with ACTIVE transfers.
It allows to filter by VO, source SE and destination SE
## fts-rest-delete-submit
This command can be used to submit a deletion job to FTS3. It supports simple and bulk submissions.
### Usage
Usage: fts-rest-snapshot [options]
Usage: fts-rest-delete-submit [options] SURL1 [SURL2] [SURL3] [...]
### Options
-h/--help
......@@ -138,34 +140,88 @@ Usage: fts-rest-snapshot [options]
--access-token
: Oauth2 access token (supported only by some endpoints, takes precedence)
--vo
: Filter by vo
-b/--blocking
: Blocking mode. Wait until the operation completes.
--source
: Filter by source se
-i/--interval
: Interval between two poll operations in blocking mode.
--destination
: Filter by destination se
-e/--expire
: Expiration time of the delegation in minutes.
--job-metadata
: Transfer job metadata.
--file-metadata
: File metadata.
-S/--spacetoken
: The space token or its description.
--dry-run
: Do not send anything, just print the json message.
-f/--file
: Name of configuration file
--retry
: Number of retries. If 0, the server default will be used. If negative, there will be no retries.
--cloud-credentials
: Use cloud credentials for the job (i. E. Dropbox).
### Example
```
$ fts-rest-snapshot -s https://fts3-devel.cern.ch:8446
Source: gsiftp://whatever
Destination: gsiftp://whatnot
VO: dteam
Max. Active: 5
Active: 1
Submitted: 0
Finished: 0
Failed: 0
Success ratio: -
Avg. Throughput: -
Avg. Duration: -
Avg. Queued: 0 seconds
Most frequent error: -
$ fts-rest-delete-submit -s https://fts3-devel.cern.ch:8446 gsiftp://source.host/file1 gsiftp://source.host/file2
Job successfully submitted.
Job id: 9fee8c1e-c46d-11e3-8299-02163e00a17a
$ fts-rest-delete-submit -s https://fts3-devel.cern.ch:8446 -f bulk.list
Job successfully submitted.
Job id: 9fee8c1e-c46d-11e3-8299-02163e00a17a
```
## fts-rest-server-status
Use this command to check on the service status.
### Usage
Usage: fts-rest-server-status [options]
### Options
-h/--help
: Show this help message and exit
-v/--verbose
: Verbose output.
-s/--endpoint
: Fts3 rest endpoint.
-j/--json
: Print the output in json format.
--key
: The user certificate private key.
--cert
: The user certificate.
--capath
: Use the specified directory to verify the peer
--insecure
: Do not validate the server certificate
--access-token
: Oauth2 access token (supported only by some endpoints, takes precedence)
-H/--host
: Limit the output to a given host
--is-active
: The tool will return < 0 on error, 0 if nothing is active, 1 if there are active transfers, 2 if there are active staging, 3 if there are both
## fts-rest-transfer-cancel
This command can be used to cancel a running job. It returns the final state of the canceled job.
Please, mind that if the job is already in a final state (FINISHEDDIRTY, FINISHED, FAILED),
......@@ -277,51 +333,6 @@ Submission time: 2014-04-15T07:07:33
Priority: 3
VO Name: atlas
```
## fts-rest-whoami
This command exists for convenience. It can be used to check, as the name suggests,
who are we for the server.
### Usage
Usage: fts-rest-whoami [options]
### Options
-h/--help
: Show this help message and exit
-v/--verbose
: Verbose output.
-s/--endpoint
: Fts3 rest endpoint.
-j/--json
: Print the output in json format.
--key
: The user certificate private key.
--cert
: The user certificate.
--capath
: Use the specified directory to verify the peer
--insecure
: Do not validate the server certificate
--access-token
: Oauth2 access token (supported only by some endpoints, takes precedence)
### Example
```
$ fts-rest-whoami -s https://fts3-pilot.cern.ch:8446
User DN: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=saketag/CN=678984/CN=Alejandro Alvarez Ayllon
VO: dteam
VO: dteam/cern
Delegation id: 9a4257f435fa2010
```
## fts-rest-transfer-status
This command can be used to check the current status of a given job
......@@ -471,7 +482,10 @@ Usage: fts-rest-transfer-submit [options] SOURCE DESTINATION [CHECKSUM]
: The source space token or its description.
-K/--compare-checksum
: Compare checksums between source and destination.
: Deprecated: compare checksums between source and destination.
-C/--checksum-mode
: Compare checksums in source, target, both or none.
--copy-pin-lifetime
: Pin lifetime of the copy in seconds.
......@@ -479,6 +493,9 @@ Usage: fts-rest-transfer-submit [options] SOURCE DESTINATION [CHECKSUM]
--bring-online
: Bring online timeout in seconds.
--timeout
: Transfer timeout in seconds.
--fail-nearline
: Fail the transfer is the file is nearline.
......@@ -518,11 +535,12 @@ Job successfully submitted.
Job id: 9fee8c1e-c46d-11e3-8299-02163e00a17a
```
## fts-rest-delete-submit
This command can be used to submit a deletion job to FTS3. It supports simple and bulk submissions.
## fts-rest-whoami
This command exists for convenience. It can be used to check, as the name suggests,
who are we for the server.
### Usage
Usage: fts-rest-delete-submit [options] SURL1 [SURL2] [SURL3] [...]
Usage: fts-rest-whoami [options]
### Options
-h/--help
......@@ -552,85 +570,13 @@ Usage: fts-rest-delete-submit [options] SURL1 [SURL2] [SURL3] [...]
--access-token
: Oauth2 access token (supported only by some endpoints, takes precedence)
-b/--blocking
: Blocking mode. Wait until the operation completes.
-i/--interval
: Interval between two poll operations in blocking mode.
-e/--expire
: Expiration time of the delegation in minutes.
--job-metadata
: Transfer job metadata.
--file-metadata
: File metadata.
-S/--spacetoken
: The space token or its description.
--dry-run
: Do not send anything, just print the json message.
-f/--file
: Name of configuration file
--retry
: Number of retries. If 0, the server default will be used. If negative, there will be no retries.
--cloud-credentials
: Use cloud credentials for the job (i. E. Dropbox).
### Example
```
$ fts-rest-delete-submit -s https://fts3-devel.cern.ch:8446 gsiftp://source.host/file1 gsiftp://source.host/file2
Job successfully submitted.
Job id: 9fee8c1e-c46d-11e3-8299-02163e00a17a
$ fts-rest-delete-submit -s https://fts3-devel.cern.ch:8446 -f bulk.list
Job successfully submitted.
Job id: 9fee8c1e-c46d-11e3-8299-02163e00a17a
$ fts-rest-whoami -s https://fts3-pilot.cern.ch:8446
User DN: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=saketag/CN=678984/CN=Alejandro Alvarez Ayllon
VO: dteam
VO: dteam/cern
Delegation id: 9a4257f435fa2010
```
## fts-rest-server-status
Use this command to check on the service status.
### Usage
Usage: fts-rest-server-status [options]
### Options
-h/--help
: Show this help message and exit
-v/--verbose
: Verbose output.
-s/--endpoint
: Fts3 rest endpoint.
-j/--json
: Print the output in json format.
--key
: The user certificate private key.
--cert
: The user certificate.
--capath
: Use the specified directory to verify the peer
--insecure
: Do not validate the server certificate
--access-token
: Oauth2 access token (supported only by some endpoints, takes precedence)
-H/--host
: Limit the output to a given host
--is-active
: The tool will return < 0 on error, 0 if nothing is active, 1 if there are active transfers, 2 if there are active staging, 3 if there are both
% FTS-REST-CLI(1) fts-rest-ban
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-ban
......
% FTS-REST-CLI(1) fts-rest-delegate
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-delegate
......@@ -45,6 +45,9 @@ This command can be used to (re)delegate your credentials to the FTS3 server
-f/--force
: Force the delegation
-H/--hours
: Duration of the delegation in hours (default: 12)
# EXAMPLE
```
$ fts-rest-delegate -s https://fts3-devel.cern.ch:8446
......
% FTS-REST-CLI(1) fts-rest-delete-submit
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-delete-submit
......
% FTS-REST-CLI(1) fts-rest-server-status
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-server-status
......
% FTS-REST-CLI(1) fts-rest-snapshot
% fts-devel@cern.ch
% August 24, 2016
# NAME
fts-rest-snapshot
# SYNOPIS
Usage: fts-rest-snapshot [options]
# DESCRIPTION
This command can be used to retrieve the internal status FTS3 has on all pairs with ACTIVE transfers.
It allows to filter by VO, source SE and destination SE
# OPTIONS
-h/--help
: Show this help message and exit
-v/--verbose
: Verbose output.
-s/--endpoint
: Fts3 rest endpoint.
-j/--json
: Print the output in json format.
--key
: The user certificate private key.
--cert
: The user certificate.
--capath
: Use the specified directory to verify the peer
--insecure
: Do not validate the server certificate
--access-token
: Oauth2 access token (supported only by some endpoints, takes precedence)
--vo
: Filter by vo
--source
: Filter by source se
--destination
: Filter by destination se
# EXAMPLE
```
$ fts-rest-snapshot -s https://fts3-devel.cern.ch:8446
Source: gsiftp://whatever
Destination: gsiftp://whatnot
VO: dteam
Max. Active: 5
Active: 1
Submitted: 0
Finished: 0
Failed: 0
Success ratio: -
Avg. Throughput: -
Avg. Duration: -
Avg. Queued: 0 seconds
Most frequent error: -
```
% FTS-REST-CLI(1) fts-rest-transfer-cancel
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-transfer-cancel
......
% FTS-REST-CLI(1) fts-rest-transfer-list
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-transfer-list
......
% FTS-REST-CLI(1) fts-rest-transfer-status
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-transfer-status
......
% FTS-REST-CLI(1) fts-rest-transfer-submit
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-transfer-submit
......@@ -72,10 +72,7 @@ format is as follows:
: Do not validate the server certificate
--access-token
: IAM-XDC access token (supported only by some endpoints, takes precedence)
--target-qos
: the target QoS that the destination has to have when the operation is finished
: Oauth2 access token (supported only by some endpoints, takes precedence)
-b/--blocking
: Blocking mode. Wait until the operation completes.
......@@ -117,16 +114,16 @@ format is as follows:
: Deprecated: compare checksums between source and destination.
-C/--checksum-mode
: Compare checksums in source, target, both or none.
: Compare checksums in source, target, both or none.
--copy-pin-lifetime
: Pin lifetime of the copy in seconds.
--bring-online
: Bring online timeout in seconds.
: Bring online timeout in seconds.
--timeout
: Transfer timeout in seconds.
: Transfer timeout in seconds.
--fail-nearline
: Fail the transfer is the file is nearline.
......@@ -155,9 +152,6 @@ format is as follows:
--ipv6
: Force ipv6
# EXAMPLE
```
$ fts-rest-transfer-submit -s https://fts3-devel.cern.ch:8446 gsiftp://source.host/file gsiftp://destination.host/file
......
% FTS-REST-CLI(1) fts-rest-whoami
% fts-devel@cern.ch
% August 24, 2016
% May 21, 2019
# NAME
fts-rest-whoami
......
......@@ -55,6 +55,12 @@ That's the purpose of a Context.
context = fts3.Context(endpoint, ucert, ukey, verify=True)
```
The endpoint to use corresponds to the FTS instance REST server and it must have the following format:
`https://<host>:<port>`
for instance https://fts3.cern.ch:8446
If you are using a proxy certificate, you can either specify only user_certificate, or point both parameters
to the proxy.
......@@ -118,7 +124,7 @@ Deserialized JSON message returned by the server (list of jobs)
#### Example:
```python
fts3.list_jobs(context. user_dn=None, vo=None)
fts3.list_jobs(context, user_dn=None, vo=None)
```
```json
......@@ -241,6 +247,208 @@ fts3.get_job_status(context, '1234-5678-abcdef', list_files=False)
}
```
### get_jobs_statuses
Get the status of a list of jobs
#### Args:
* **context** fts3.rest.client.context.Context instance
* **job_ids** A list of job IDs
* **list_files** If True, the status of each individual file will be queried
#### Returns:
Deserialized JSON message returned by the server containing a list of jobs plus optionally a list of the following fields for each transfer part of the job:
file_state, dest_surl, finish_time, start_time, reason, source_surl, file_metadata
#### Example:
```python
fts3.get_jobs_statuses(context, ['e360232c-7ba6-11e9-aea1-02163e00a077', 'e3941092-7ba6-11e9-8851-02163e00a077', 'e3b1d898-7ba6-11e9-8851-02163e00a077'], list_files=True)
```
```json
[
{
'cred_id': 'bfd3b0c180c099ea',
'user_dn': '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amanzi/CN=683749/CN=Andrea Manzi',
'job_type': 'N',
'retry': -1,
'job_id': 'e360232c-7ba6-11e9-aea1-02163e00a077',
'cancel_job': False,
'job_state': 'SUBMITTED',
'submit_host': 'fts3devel02.cern.ch',
'priority': 3,
'source_space_token': '',
'max_time_in_queue': None,
'job_metadata': {
'test': 'test_multiple_jobs',
'label': 'fts3-tests'
},
'source_se': None,
'files': [
{
'dest_surl': 'mock://test.cern.ch/ttqv/pryb/nnvw?size_post=1048576&time=2',
'finish_time': None,
'start_time': None,
'file_metadata': None,
'reason': None,
'source_surl': 'mock://somewhere.uk/qtgc/mxar/aohc?size=1048576',
'file_state': 'SUBMITTED'
},
{
'dest_surl': 'mock://test.cern.ch/swnx/jznu/laso?size_post=1048576&time=2',
'finish_time': None,
'start_time': None,
'file_metadata': None,
'reason': None,
'source_surl': 'mock://aplace.es/vgcl/cpwh/umfo?size=1048576',
'file_state': 'SUBMITTED'
},
{
'dest_surl': 'mock://test2.cern.ch/kzqa/pvhf/jmto?size_post=1048576&time=2',
'finish_time': None,
'start_time': None,
'file_metadata': None,
'reason': None,
'source_surl': 'mock://test2.cern.ch/yasr/anan/zhbz?size=1048576',
'file_state': 'SUBMITTED'
}
],
'http_status': '200 Ok',
'bring_online': -1,
'reason': None,
'space_token': '',
'submit_time': '2019-05-21T09:00:32',
'retry_delay': 0,
'dest_se': None,
'internal_job_params': 'nostreams:1',
'vo_name': 'dteam',
'copy_pin_lifetime': -1,
'verify_checksum': 'n',
'job_finished': None,
'overwrite_flag': False
},
{
'cred_id': 'bfd3b0c180c099ea',
'user_dn': '/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=amanzi/CN=683749/CN=Andrea Manzi',
'job_type': 'N',
'retry': -1,
'job_id': 'e3941092-7ba6-11e9-8851-02163e00a077',
'cancel_job': False,
'job_state': 'SUBMITTED',
'submit_host': 'fts3devel02.cern.ch',
'priority': 3,
'source_space_token': '',
'max_time_in_queue': None,
'job_metadata': {
'test': 'test_multiple_jobs',
'label': 'fts3-tests'
},
'source_se': None,
'files': [
{
'dest_surl': 'mock://test2.cern.ch/plym/iotw/awln?size_post=1048576&time=2',
'finish_time': None,
'start_time': None,
'file_metadata': None,
'reason': None,
'source_surl': 'mock://aplace.es/dlgj/tvlr/kyqh?size=1048576',
'file_state': 'SUBMITTED'