Open BackendVFS ObjectStore files in R/W mode when obtaining exclusive locks
Summary
BackendVFS always opens files in RO mode when obtaining locks. This is typically OK for flock-style locks, but some FS (in particular NFS) implementations emulate flock-style locks with fcntl locks, which require a RW handle to obtain an exclusive lock.
Steps to reproduce
- Configure CTA to use a mounted NFS dir as its ObjectStore
- Try to initialize CTA ObjectStore
(This might be as simple as running cta-objectstore-initialize $NFS_MNT_DIR
)
What is the current bug behavior?
CTA ObjectStore initialization fails with In BackendVFS::lockHelper(): Failed to flock file /mnt/nfs/objectstore/.root.lock: Bad file descriptor
What is the expected correct behavior?
CTA ObjectStore should initialize successfully
Relevant logs and/or screenshots
[cta@fermicloud524 orchestration]$ ./create_instance.sh -n cta -b ~ -B CTA-build -D -O -d database.yaml;
...
pod/init created
Waiting for init..........................................................................................................................................................................................................................init pod in Error status here are its last log lines:
...
ERROR: Could not Wipe the objectstore. cta-objectstore-initialize /mnt/nfs/objectstore FAILED
ERROR: init pod in Error state. Initialization failed.
[cta@fermicloud524 orchestration]$ kubectl logs init -n cta
...
Using this configuration for library:
Configuring mhvtl library
DRIVESLOT is not defined, using driveslot 0
export LIBRARYTYPE=mhvtl
export LIBRARYNAME=VLSTK10
export LIBRARYDEVICE=sg2
export DRIVENAMES=(VDSTK11 VDSTK12 VDSTK13)
export DRIVEDEVICES=(nst1 nst2 nst0)
export TAPES=(V01001 V01002 V01003 V01004 V01005 V01006 V01007)
export driveslot=0
Configuring objectstore:
Configuring file objectstore
Wiping objectstore
Failed to initialise the root entry in a new cta::objectstore::BackendVFS objectstore
In BackendVFS::lockHelper(): Failed to flock file /mnt/nfs/objectstore/.root.lock: Bad file descriptor
/lib64/libctacommon.so.0(cta::exception::Backtrace::Backtrace(bool)+0x6e) [0x7f38f8d9c3bc]
/lib64/libctacommon.so.0(cta::exception::Exception::Exception(std::string const&, bool)+0x89) [0x7f38f8d9dc53]
/lib64/libctaobjectstore.so.0(cta::objectstore::BackendVFS::lockHelper(std::string, int, unsigned long)+0x782) [0x7f38f95457a2]
/lib64/libctaobjectstore.so.0(cta::objectstore::BackendVFS::lockExclusive(std::string, unsigned long)+0x53) [0x7f38f9545b5d]
cta-objectstore-initialize(cta::objectstore::ScopedExclusiveLock::lock(cta::objectstore::ObjectOpsBase&, unsigned long)+0xa1) [0x41d459]
cta-objectstore-initialize(cta::objectstore::ScopedExclusiveLock::ScopedExclusiveLock(cta::objectstore::ObjectOpsBase&, unsigned long)+0x59) [0x41d2d1]
cta-objectstore-initialize(main+0x2f7) [0x418a1a]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f38f6903555]
cta-objectstore-initialize() [0x4180c9]
ERROR: Could not Wipe the objectstore. cta-objectstore-initialize /mnt/nfs/objectstore FAILED
Possible fixes
BackendVFS.cpp can be modified to open files RO or RW based on the type of lock requested. The relevant code is here: https://gitlab.cern.ch/cta/CTA/-/blob/master/objectstore/BackendVFS.cpp#L257
I have a fix for this implemented, will link a pull request to this issue.
Open to suggestions for how to implement a unit test for this change, but it doesn't seem to fit into existing infrastructure (https://gitlab.cern.ch/cta/CTA/-/blob/master/objectstore/BackendTest.cpp#L61) cleanly, and since BackendVFS is not used in production anyway I'm not sure it's worth the effort.