Add test that file has been deleted from FST

In branch fst_delete_on_close, CTA/continuousintegration/orchestration/tests/delete_on_closew_error.sh.

This script tests the EOS delete on close workflow. It tests that the file has been removed from the namespace (using eos fileinfo). There should also be a test that the physical file was removed from the FST.

assigned to @afonso

added Continuous integration in CTA-old label

This one is a perfect candidate for eos-report where we can track if a file has been deleted on the FSTs.

My idea is more to track file deletion in eos-report and inspect the namespace and FST disks after a test to see if we have some left over.

• First point: go for what we have eos-report and file replicas left in FST.
• Second point: look for leaked attributes in the namespace using eos-ns-inspect.

removed milestone

added Code Quality in CTA-old label

removed Continuous integration in CTA-old label

assigned to @afonso and unassigned @jleduc

added workflowAssigned in CTA-old label

We have been having some problems with understanding delete on close behaviour (see https://gitlab.cern.ch/cta/operations/-/issues/607). It would be good to improve the test coverage of this feature in the CI.

I have added a new test on https://gitlab.cern.ch/cta/CTA/-/commits/fst_delete_on_close for the scenario where the checksum fails, but the file is still queued for archival (unwanted behaviour).

This is similar to the issue reported on https://gitlab.cern.ch/cta/operations/-/issues/614.

mentioned in merge request !93 (merged)

We will split these new tests in two parts:

1. Testing delete-on-close "good day" case, when storage_class=fail_on_closew_test is used.

• We will not need to check directly with the FST that the physical copy of the file has been deleted. This check, besides being considerably more complex to perform, should not be necessary because the namespace entry is only removed once all the replicas are deleted from the disks/tape. Therefore, checking that the namespace entry was removed should be enough to guarantee that there are no copies.
• This has has been added in this commit: https://gitlab.cern.ch/cta/CTA/-/commit/98d0a7eefa6567a630e945e1bbb5443aff5633c1 and this merge request: !93 (merged)

2. Testing delete-on-close after receiving a wrong checksum:

• As mentioned on this ticket https://gitlab.cern.ch/cta/operations/-/issues/614, this is currently not working because the archival request is still queued even though the delete-on-close event is triggered.
• This test will stay in the branch https://gitlab.cern.ch/cta/CTA/-/commits/fst_delete_on_close, waiting for the fix to be done before merging.

Hi @afonso, even if the archival request is being triggered on close, shouldn't the test still work? The disk replica will be deleted and a DELETE event will be generated. This will remove the extra archival request.

Sometimes there is racy behaviour where the DELETE event can be sent before the CLOSEW event. This will generate an archive error in the tapeserver but the delete on close part still works.

Is it therefore not possible to merge this test without waiting for the other issue to be fixed?

Hi @mdavis , I am not sure if I understood all, but for the moment the test with the wrong checksum does not pass. xrdcp fails and returns an error - as expected - but the new namespace entry remains while it should have been deleted (I think).

added workflowIn Progress in CTA-old label and removed workflowAssigned in CTA-old label

added workflowBlocked in CTA-old label and removed workflowIn Progress in CTA-old label

added Continuous Integration + 1 deleted label

added prioritylow label

I'm removing myself from this issue for the moment, due to the low priority.

added 1 deleted label and removed 1 deleted label

unassigned @afonso

changed milestone to %CI update push

mentioned in commit 14551b3b

mentioned in commit a935539b

mentioned in commit 0e931188

mentioned in commit 4d5303ba

added ci: tests label