Add setter/getter for encryption key name in cta-admin tapepool

Problem

At the moment there is an ENCRYPTION_KEY_NAME field in the TAPE_POOL table of the CTA Catalogue, but we do not have the logic to set, get or use this value inside the CTA code:

• https://gitlab.cern.ch/cta/cta-catalogue-schema/-/blob/ae7671e0/common_catalogue_schema.sql#L121
• #137 (closed)

Instead, we rely on the cta-get-encryption-key-puppet.py script to return the encryption key name based on the tape pool name:

• https://gitlab.cern.ch/cta/operations/-/issues/1613#note_9151566

Expected solution

• Add the option to get/set the ENCRYPTION_KEY_NAME value on the tape pool with the cta-admin tp command. Modify CTA to use this value when archiving files to new tapes.
• Deprecate the usage of the option --pool-name test_encryption from the cta-get-encryption-key-puppet.py script.
• Allow CTA to directly key the encryption key from the .json file.

References

CTA Ops docs on encryption:

• https://tapeoperations.docs.cern.ch/ctaops/data_encryption_and_permissions/cta_encryption_logic/

added Needs Discussion cta: Tape server priorityhigh typefeature workflowin progress labels

assigned to @afonso

changed the description

marked this issue as related to #137 (closed)

created branch 1104-add-setter-getter-for-encryption-key-name-in-cta-admin-tapepool to address this issue

mentioned in merge request !833 (merged)

Some open questions for discussion:

1

The TAPE_POOL table already contains a IS_ENCRYPTED column (boolean).
How should we use this together with the ENCRYPTION_KEY_NAME column?

Possible approaches:

1. Deprecate IS_ENCRYPTED. The tape pool is encrypted if (and only if) ENCRYPTION_KEY_NAME is not null.
2. The tape pool is encrypted if both IS_ENCRYPTED and ENCRYPTION_KEY_NAME are true and not null.

2

Should we simply stop using the option --pool-name <value> from the cta-get-encryption-key-puppet.py script?

• https://gitlab.cern.ch/cta/CTA/-/blob/main/tapeserver/castor/tape/tapeserver/daemon/EncryptionControl.cpp#L79

We can safely remove the option from the python script after this change is released with CTA.

3

At the moment, if a non-encrypted tape is mounted for writing with an associated encrypted tape pool, it seems that CTA will simply set the new encryption key without regard for any data that may have been written before without encryption:

• https://gitlab.cern.ch/cta/CTA/-/blob/main/tapeserver/castor/tape/tapeserver/daemon/EncryptionControl.cpp#L101

• Is this really a problem?

  • I would expect so, unless there is another protection in place that I'm not aware of.

• Should we prevent any tapes with (non-encrypted) data from having their encryption key set?

4

Should we optionally allow CTA to directly get the encryption key from the .json file?
How should it distinguish between a script and a json file?

Possible approaches:

1. Look at file type. Only support .sh/.py/.json.
2. Create a new configuration.
3. Do not read directly from .json.

Topics agreed:

1. Yes, we will stop using IS_ENCRYPTED. To be seen if we will delete this column from the catalogue or not (probably only in a future catalogue release).
2. Yes, remove --pool-name argument. This will not longer be necessary and only adds extra complexity.
3. This is not a problem. The tape drives should still be able to read non-encrypted files in encrypted tapes.
4. No. We want a script that is able to get keys from vault or other key storage systems, and be able to manage authentication tokens.
5. (Extra) We should net allow the encryption key of a tape to be modified once it has been set.

removed Needs Discussion label

mentioned in issue #1116 (closed)

There is the question on how to allow the transition between IS_ENCRYPTED and ENCRYPTION_KEY_NAME without the risk of missing the encryption of some files.

This may happen if we deploy the new version and a new tape is selected before ENCRYPTION_KEY_NAME is set for its tape pool. This tape could end up containing some non-encrypted files even if IS_ENCRYPTED was true.

To avoid this, we probably should support both ENCRYPTION_KEY_NAME and IS_ENCRYPTED for a while (probably until the next catalogue release is done):

• If ENCRYPTION_KEY_NAME is set we ignore IS_ENCRYPTED.
• If ENCRYPTION_KEY_NAME is not set, we check and use IS_ENCRYPTED as before.

In the future catalogue release 16.0 we could remove the column IS_ENCRYPTED in the following way:

1. The migration script starts by validating that all the columns with IS_ENCRYPTED have a non-null ENCRYPTION_KEY_NAME. If not, it should fail.
2. The SQL queries that use IS_ENCRYPTED should only use it if the CTA Catalogue is version 15.0. Otherwise IS_ENCRYPTED should not be queried and instead be replaced by a dummy value (or ENCRYPTION_KEY_NAME IS NOT NULL). This is only necessary for the pivot release.
3. All other releases can ignore the IS_ENCRYPTED column because it is no longer expected in version 16.0`.

In the meanwhile, we could enforce IS_ENCRYPTED not to be used by modifying the get encryption script.

As discussed with @jleduc , we will keep both IS_ENCRYPTED and ENCRYPTION_KEY_NAME, but IS_ENCRYPTED will no longer be set directly. Instead, it will be set based on the new value of ENCRYPTION_KEY_NAME:

Instead:

• If ENCRYPTION_KEY_NAME is NULL --> IS_ENCRYPTED set to false
• If ENCRYPTION_KEY_NAME not NULL --> IS_ENCRYPTED set to true

In this case, IS_ENCRYPTED will be kept until the next catalogue release as a way to guarantee backward compatibility with the current tape pool configurations in the CTA Catalogue, and allow for a smooth transition to ENCRYPTION_KEY_NAME.

added workflowtesting label and removed workflowin progress label

mentioned in issue #1121 (closed)

closed with merge request !833 (merged)

mentioned in commit 972a11a2

mentioned in commit 6096ecb6

mentioned in merge request !861 (merged)