mounting Ceph volumes (on MGM) to support direct writing via CephFS
Andreas is working on a "file adoption" feature https://eos-community.web.cern.ch/t/direct-access-to-eos-data-on-cephfs-in-k8s/734/4 that would allow clients to write directly into a CephFS scratch area, then a FST would perform stage-in to EOS instantly via hardlinking. This should be optimal for performance.
However for this to work, we would need a way to mount cephfs on FSTs differently, so that a FST can see the rest of the cephfs filesystem, not only its own storage directory. And the FST would need to be configured to only use its own subdirectory. So if the volume mount was /cephfs instead of /fst_storage , the FST would need to be configured to use some subdirectory like e.g. /cephfs/eos-data/fst01.
There are ideas about how each pod in a statefulset could know its index: https://github.com/kubernetes/kubernetes/issues/40651
But in the meantime it could be done using the name of each FST pod, perhaps via an env var, e.g. like https://stackoverflow.com/questions/50750672/is-there-a-way-to-get-ordinal-index-of-a-pod-with-in-kubernetes-statefulset-conf
env:
- name: fst_name
valueFrom:
fieldRef:
fieldPath: metadata.name
Then the FST could use some environment variables to detect the correct location, e.g. /cephfs/$configurable_prefix/$fst_name
Perhaps it could be considered as a potential longer term improvement.