`cta-taped` behavior when tape cannot be mounted
Contrary to the current CI test cases, we regularly have to face non mountable tapes in production. It happens if a tape is stuck in another tape drive, fell on the floor,... These cases are handled by TAS at CERN: it catch escaped tapes, mark them as DISABLED and open a ticket.
Because of this partial CERN solution, cta-taped
has never been tested under this kind of scenarios and we need to understand how cta-taped
reacts to various tape missing cases.
cta-rmcd
cannot mount tape: volume in use
DISABLE
all tapes except one that is mounted in the second tape drive set in DOWN
state.
First quick result
cta-rmcd
fails to mount the tape:
04/02 10:49:08 85 rmc_srv_mount: RMC92 - mount request by 1000,33 from localhost
04/02 10:49:08 85 rmc_srv_mount: RMC98 - mount V00101/0 on drive 0
04/02 10:49:08 85 rmc_sendrep: smc_mount: Asked for V00101, got reply for V00101
04/02 10:49:08 85 rmc_sendrep: smc_mount: Location: data transfer element (0x4)
04/02 10:49:08 85 rmc_sendrep: smc_mount: SR018 - mount of V00101 on drive 0 failed : volume in use
04/02 10:49:08 85 rmc_srv_mount: returns 2204
Return code is 2204
, but as the tape was about to be mounted cta-taped
started a tape session:
- it prefetched some files from eoscta instance
-
TapeWrite
thread complains thatMSG="Failed to mount the tape for read/write access"
- It even says:
MSG="TapeWriteSingleThread: No tape to unload"
when failing the tape write session.
Then end of session:
Apr 2 10:54:09.447826 tpsrv01 cta-taped: LVL="INFO" PID="266" TID="266" MSG="Tape session finished" drive_name="VDSTK01" instance="CI" sched_backend="VFS" tapeVid="V00101" moun
tType="ArchiveForUser" mountId="2" volReqId="2" vendor="vendor" vo="vo" mediaType="T10K500G" tapePool="ctasystest" logicalLibrary="VDSTK01" capacityInBytes="500000000000" status
="failure" wasTapeMounted="0" mountTime="0.000000" positionTime="0.000000" waitInstructionsTime="0.124154" waitFreeMemoryTime="0.000000" waitDataTime="0.000000" waitReportingTim
e="0.267970" checksumingTime="0.000000" readWriteTime="0.000000" flushTime="0.000000" unloadTime="0.000000" unmountTime="0.000000" encryptionControlTime="0.000001" transferTime=
"0.392124" totalTime="300.925606" deliveryTime="300.925606" drainingTime="0.000000" dataVolume="0" filesCount="0" headerVolume="0" payloadTransferSpeedMBps="0.000000" driveTrans
ferSpeedMBps="0.000000" Error_tapeMountForWrite="1" tapeDrive="VDSTK01" pid="686" exitCode="0" killSignal="0"
But the problem appears later: No tape is mounted but the archive queue on this tapepool is drained after 3 tries like this one.