method for mounting a cephfs filesystem
At first I was confused about why PVCs are used to mount CephFS, as this does not necessarily provide any guarantees (capacity or exclusive access) or benefits compared to directly mounting CephFS in a pod without a PVC.
In our environment we have one statically provisioned CephFS filesystem. It is from Manila but I would rather not have to use Manila CSI. Besides, our clusters are deployed using the IaaC approach and may be destroyed and rebuilt from scratch from time to time. This is very useful as a fallback or disaster recovery mechanism, but can make use of PVCs a bit cumbersome as we have to statically create and define all PVs.
In any case, what I end up with when using PVCs this way is all FSTs mounting the same CephFS share and directory, and failing to boot:
[root@eos-mgm-0 /]# eos fs ls -l
┌──────────────────────────────────────────┬────┬──────┬────────────────────────────────────┬────────────────────────────────┬────────────────┬──────────┬────────────┬──────────────┬────────────┬──────┬────────┬──────────────┬────────────────┬────────────────────────┐
│host │port│ id│ uuid│ path│ schedgroup│ headroom│ boot│ configstatus│ drain│ usage│ active│ scaninterval│ health│ statuscomment│
└──────────────────────────────────────────┴────┴──────┴────────────────────────────────────┴────────────────────────────────┴────────────────┴──────────┴────────────┴──────────────┴────────────┴──────┴────────┴──────────────┴────────────────┴────────────────────────┘
eos-fst-0.eos-fst.eos.svc.kermes-dev.local 1095 1 7cd12a93-531b-44f1-b875-297f7af8d3ba /fst_storage default.0 0.00 bootfailure rw nodrain 0.00 online 604800 N/A
eos-fst-1.eos-fst.eos.svc.kermes-dev.local 1095 2 1304182b-876e-410a-a619-6262380b8f48 /fst_storage default.1 0.00 bootfailure rw nodrain 0.00 online 604800 N/A
eos-fst-2.eos-fst.eos.svc.kermes-dev.local 1095 3 6bebc747-b57e-4709-a956-aafc71ad4772 /fst_storage default.2 0.00 bootfailure rw nodrain 0.00 online 604800 N/A
eos-fst-3.eos-fst.eos.svc.kermes-dev.local 1095 4 660b89f4-fd3e-474b-85ec-7e35958e0624 /fst_storage default.3 0.00 down rw nodrain 8.53 offline 604800 N/A
[root@eos-mgm-0 /]# eos fs status 1| grep errmsg
stat.errmsg := filesystem has a different label (fsid=1, uuid=7cd12a93-531b-44f1-b875-297f7af8d3ba) than the configuration
[root@eos-mgm-0 /]# eos fs status 2| grep errmsg
stat.errmsg := filesystem has a different label (fsid=2, uuid=1304182b-876e-410a-a619-6262380b8f48) than the configuration
[root@eos-mgm-0 /]# eos fs status 3| grep errmsg
stat.errmsg := filesystem has a different label (fsid=3, uuid=6bebc747-b57e-4709-a956-aafc71ad4772) than the configuration
So all the FST pods see the same location with this unique file.
[root@eos-fst-2 /]# cat /fst_storage/.eosfsuuid
42f67649-697f-412d-974e-ce93c38f1435
If I understand correctly each FST needs independent storage - although I recall discussion about a unified layout, and it would be highly desirable if the FSTs could serve requests interchangeably instead of each FST owning an independent storage location. Not sure if there is some development towards that?
Using one single cephfs filesystem is also what I had in mind to make this idea possible for our compute jobs: https://eos-community.web.cern.ch/t/direct-access-to-eos-data-on-cephfs-in-k8s/734/3