Revert eviction test to original behaviour and retry failed requests
UPDATE: After taking a look at the code of the eviction test there was a misunderstanding of xargs flags usage, so for now we will just revert to the original behaviour and see if this improves the situation. For further details see: #662 (closed)
Original Title: Retry failed evict requests in system tests for 'Resource temporarily unavailable' errors
Problem to solve
One of the main causes for client system tests failing lately is that some files not being evicted due to the following errors:
MGM
240402 13:54:14 ERROR [12001/01200] poweruser1 ::triggerPrepareWorkflow Unable to prepare - synchronous prepare workflow error /eos/ctaeos/preprod/1cf1bf0a-9bc5-4aa5-a7f8-41ec31f370a8/0/04359; Unable to trigger workflow - synchronous workflow failed /eos/ctaeos/preprod/1cf1bf0a-9bc5-4aa5-a7f8-41ec31f370a8/0/04359; Resource temporarily unavailable
WFE
240402 13:54:14 INFO WFE:1620 default SYNC::EVICT_PREPARE /eos/ctaeos/preprod/1cf1bf0a-9bc5-4aa5-a7f8-41ec31f370a8/0/04359 ctafrontend:10955 fxid=0000111d mgm.reqid="*"
240402 13:54:14 INFO WFE:2182 fxid=111d file=/eos/ctaeos/preprod/1cf1bf0a-9bc5-4aa5-a7f8-41ec31f370a8/0/04359 msg="Failed to issue evict for evict_prepare event"
Proposal
Track the failed requests and retry those that matched the error one second after all the requests have been submitted.
Edited by Pablo Oliver Cortes