Kubernetes 挂 CephFS 的三种方式：直挂、静态 PV/PVC、动态分配-星海夜航

CephFS 是跟 POSIX 兼容的文件系统，直接提供 Ceph 存储集群上的文件访问，从 Jewel 版本（10.2.0）开始稳定。它要求至少跑一个元数据服务 MDS（ceph-mds），由 MDS 管理文件元数据并协调对存储的访问。做共享读写的数据卷时，文件存储比对象存储划算——对象存储要专门软件和大容量盘，数据量不大、只需要多 Pod 共享读写，就没必要上对象存储。

下面三种挂载方式按由浅到深排：Pod 里直接写 cephfs 卷、PV/PVC 静态绑定、cephfs-provisioner 动态分配。前提是先有可用的 Ceph 集群和 CephFS，并且 Kubernetes 所有节点（尤其 master）都装过客户端：

yum install -y ceph-common

还要把 ceph.conf 放到每个节点的 /etc/ceph 目录下。

准备 secret

key 要在 Ceph 集群的 mon 节点上取，并且必须做 base64 编码后写进 yaml：

# 方式1：
[cephu@ceph-node1 ~]$ sudo ceph auth get-key client.admin
AQB+dXxfhmh4LRAAM3Ow+ZdP64Py1N5ZvBgKiA==

[cephu@ceph-node1 ~]$ echo -n "AQB+dXxfhmh4LRAAM3Ow+ZdP64Py1N5ZvBgKiA=="| base64
QVFCK2RYeGZobWg0TFJBQU0zT3crWmRQNjRQeTFONVp2QmdLaUE9PQ==

# 方式2：
[cephu@ceph-node1 ~]$ sudo ceph auth get-key client.admin | base64
QVFCK2RYeGZobWg0TFJBQU0zT3crWmRQNjRQeTFONVp2QmdLaUE9PQ==

#首先创建一个名称空间
[root@k8s-master1 ~]# kubectl create ns dev
namespace/dev created

#然后创建secret类型的资源文件
[root@k8s-master1 ~]# vim ceph_secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: dev
type: kubernetes.io/rbd
data:
  key: QVFCK2RYeGZobWg0TFJBQU0zT3crWmRQNjRQeTFONVp2QmdLaUE9PQ==

[root@k8s-master1 ~]# kubectl apply -f ceph_secret.yaml 
secret/ceph-secret created

方式一：Pod 直接挂 cephfs 卷

和 emptyDir 的区别是数据会不会跟着 Pod 消失：emptyDir 在删 Pod 时清空，cephfs 卷只是被卸载，数据留在集群里，而且能多设备同时读写。

[root@k8s-master1 ~]# vim myapp-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp
  namespace: dev
spec:
  containers:
  - name: myapp
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs	#指定使用的挂载卷名称
  volumes:
  - name: cephfs		#定义挂载卷名称
    cephfs:  #挂载类型为cephFS
      monitors:
      - 192.168.66.201:6789	#ceph集群mon节点
      - 192.168.66.202:6789
      - 192.168.66.203:6789
      user: admin	#ceph认证用户
      path: /
      secretRef:
        name: ceph-secret  #调用ceph认证secret

[root@k8s-master1 ~]# kubectl apply -f myapp-pod.yaml 
pod/myapp created 

[root@k8s-master1 ~]# kubectl get po -n dev
NAME    READY   STATUS    RESTARTS   AGE
myapp   1/1     Running   0          94s

方式二：静态 PV 与 PVC

PV 里除了 monitors、user、secretRef，还要指定 persistentVolumeReclaimPolicy: Delete 和 accessModes: ReadWriteMany：

[root@k8s-master1 ~]# vim myapp-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cephfs-pv
  namespace: dev
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors:
    - 192.168.66.201:6789       #ceph集群mon节点地址和端口
    - 192.168.66.202:6789
    - 192.168.66.203:6789
    user: admin		#ceph的认证用户
    path: /
    secretRef:
      name: ceph-secret		#此处使用上面创建的secret资源中定义的secret名称
    readOnly: false
  persistentVolumeReclaimPolicy: Delete

[root@k8s-master1 ~]# kubectl apply -f myapp-pv.yaml 
persistentvolume "cephfs-pv" created

[root@k8s-master1 ~]# kubectl get pv cephfs-pv -n dev

PVC 的访问模式和存储容量必须和 PV 完全对得上，否则绑定不成功，一直 Pending：

[root@k8s-master1 ~]# vim myapp-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-pvc
  namespace: dev
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

[root@k8s-master1 ~]# kubectl apply -f myapp-pvc.yaml 
persistentvolumeclaim/cephfs-pvc created 

[root@k8s-master1 ~]# kubectl get pvc cephfs-pvc -n dev

Pod 引用 PVC：

[root@k8s-master1 ~]# vim myapp-pod-pv.yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: cephfs-pv-myapp
  name: cephfs-pv-myapp
  namespace: dev
spec:
  containers:
  - name: cephfs-pv-myapp
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - mountPath: "/mnt/cephfs"	#映射至容器中路径
      name: cephfs-myapp
      readOnly: false
  volumes:
  - name: cephfs-myapp
    persistentVolumeClaim:
      claimName: cephfs-pvc     #挂载的pvc卷名

[root@k8s-master1 ~]# kubectl apply -f myapp-pod-pv.yaml 
pod/cephfs-pv-myapp created

[root@k8s-master1 ~]# kubectl get po cephfs-pv-myapp -n dev
NAME              READY   STATUS    RESTARTS   AGE
cephfs-pv-myapp   1/1     Running   0          25s

[root@k8s-master1 ~]# kubectl describe po cephfs-pv-myapp -n dev

方式三：provisioner 动态分配

Kubernetes 官方没有为 cephfs 提供类似 Ceph RBD 的 storageClass 动态分配，社区用 cephfs-provisioner 补上这个功能。要注意这个仓库已经不再维护（说明见 https://github.com/kubernetes/org/issues/1563），试验环境可以用，生产建议换当前在维护的驱动。

先建 provisioner 用的 admin-secret，名称空间单独用 cephfs：

#在ceph-mon节点查看admin用户的权限
[cephu@ceph-node1 ~]$ sudo ceph auth get-key client.admin|base64
QVFCK2RYeGZobWg0TFJBQU0zT3crWmRQNjRQeTFONVp2QmdLaUE9PQ==

#在K8S-Master节点创建名称空间cephfs
[root@k8s-master1 ~]# kubectl create ns cephfs

#在K8S-Master1节点上创建cephfs所以定义文件存储目录
[root@k8s-master1 ~]# mkdir ceph-cephfs
[root@k8s-master1 ~]# cd ceph-cephfs/

#在K8S集群Master节点上创建secret资源并使用ceph中转换后key
[root@k8s-master1 ceph-cephfs]# vim admin-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
  namespace: cephfs
type: kubernetes.io/rbd
data:
  key: QVFCK2RYeGZobWg0TFJBQU0zT3crWmRQNjRQeTFONVp2QmdLaUE9PQ==

#执行secret定义文件
[root@k8s-master1 ceph-cephfs]# kubectl apply -f admin-secret.yaml 
secret/ceph-secret-admin created

#查看secret资源信息
[root@k8s-master1 ceph-cephfs]# kubectl get secret -n cephfs
NAME                   TYPE                                  DATA   AGE
ceph-secret-admin     kubernetes.io/rbd                     1      82s

再部署 provisioner，一份 yaml 里包含 ServiceAccount、ClusterRole、ClusterRoleBinding、Role、RoleBinding 和 Deployment，关键环境变量是 PROVISIONER_NAME=ceph.com/cephfs，command 是 /usr/local/bin/cephfs-provisioner：

[root@k8s-master1 ~]# vim storage-cephfs-provisioner.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-provisioner
  namespace: cephfs
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
subjects:
  - kind: ServiceAccount
    name: cephfs-provisioner
    namespace: cephfs
roleRef:
  kind: ClusterRole
  name: cephfs-provisioner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-provisioner
  namespace: cephfs
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-provisioner
  namespace: cephfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-provisioner
subjects:
- kind: ServiceAccount
  name: cephfs-provisioner
  namespace: cephfs

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cephfs-provisioner
  namespace: cephfs
spec:
  selector:
    matchLabels:
      app: cephfs-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: cephfs-provisioner
    spec:
      containers:
      - name: cephfs-provisioner
        image: "registry.cn-chengdu.aliyuncs.com/ives/cephfs-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/cephfs
        command:
        - "/usr/local/bin/cephfs-provisioner"
        args:
        - "-id=cephfs-provisioner-1"
      serviceAccount: cephfs-provisioner

[root@k8s-master1 ceph-cephfs]# kubectl get pods -n cephfs |grep cephfs
cephfs-provisioner-6d76ff6bd5-pcz5m   1/1     Running   0          103s

StorageClass 的 provisioner 要写成上面那个名字，monitors、adminId、adminSecretName、adminSecretNamespace 四项指向已有资源；资源名后面创建 PVC 时要与之绑定，别随手改名：

[root@k8s-master1 ceph-cephfs]# vim storageclass-cephfs.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cephfs	#注意：该名称后面创建pvc时要与之绑定
  namespace: cephfs
provisioner: ceph.com/cephfs	#使用外部cephfs
parameters:
    monitors: 192.168.66.201:6789,192.168.66.202:6789,192.168.66.203:6789	#ceph集群mon节点地址和端口
    adminId: admin  #cephfs认证用户
    adminSecretName: ceph-secret-admin  #secret认证资源NAME名称
    adminSecretNamespace: "cephfs"	#secret所在的名称空间名称

[root@k8s-master1 ceph-cephfs]#kubectl apply -f storageclass-cephfs.yaml
storageclass.storage.k8s.io/cephfs created

[root@k8s-master1 ceph-cephfs]# kubectl get sc

测试 PVC 用注解 volume.beta.kubernetes.io/storage-class: "cephfs" 触发分配。状态 Bound 表示成功；Pending 或 Failed 就得 kubectl describe 看事件、查 provisioner 日志：

[root@k8s-master1 ceph-cephfs]# vim cephfs-pvc-test.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-pvc-test
  namespace:  cephfs
  annotations:
    volume.beta.kubernetes.io/storage-class: "cephfs"  #注意：填写storage资源名称，上面创建的storage资源名称为【cephfs】
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi

[root@k8s-master1 ceph-cephfs]# kubectl apply -f cephfs-pvc-test.yaml 
persistentvolumeclaim/cephfs-pvc-test created

[root@k8s-master1 ceph-cephfs]#kubectl get pvc cephfs-pvc-test -n cephfs

Pod 绑这个 PVC。两条硬规则：PVC 与 Pod 必须在同一名称空间；PVC 被删除，关联 Pod 的数据也跟着没了。

[root@k8s-master1 ceph-cephfs]# vim cephfs-pvc-pod.yaml 
kind: Pod
apiVersion: v1
metadata:
  labels:
    app: cephfs-pvc-pod
  name: cephfs-pv-pod
  namespace: cephfs
spec:
  containers:
  - name: cephfs-pv-busybox
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs-vol
      readOnly: false
  volumes:
  - name: cephfs-vol
    persistentVolumeClaim:
      claimName: cephfs-pvc-test		#Pod关联所要绑定的PVC名称

[root@k8s-master1 ceph-cephfs]# kubectl apply -f cephfs-pvc-pod.yaml 
pod/cephfs-pv-pod created

[root@k8s-master1 ceph-cephfs]# kubectl get po -n cephfs
NAME                                  READY   STATUS    RESTARTS   AGE
cephfs-pv-pod                         1/1     Running   0          43s

[root@k8s-master1 ceph-cephfs]# kubectl get po -n cephfs -o wide

验证数据是否真的持久

拿 nginx 做一轮：在 webapp 名称空间建 5Gi 的 PVC，挂到 /usr/share/nginx/html，往 index.html 写一句话，再删掉 Deployment 重建，看内容还在不在。

#创建一个名称空间，命名为webapp
[root@k8s-master1 ceph-cephfs]# kubectl create ns webapp

#pvc与pod所属名称空间都为webapp
#创建PVC
[root@k8s-master1 ceph-cephfs]# vim cephfs-pvc-nginx.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-pvc-nginx
  namespace: webapp
  annotations:
    volume.beta.kubernetes.io/storage-class: "cephfs"  #指定绑定的storage名称
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi

#创建Pod
[root@k8s-master1 ceph-cephfs]# vim cephfs-deploy-nginx.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: nginx-web
  name: nginx-web
  namespace: webapp
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: nginx-web
  template:
    metadata:
      labels:
        k8s-app: nginx-web
      namespace: webapp
      name: nginx-web
    spec:
      containers:
        - name: nginx-web
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
              name: web
              protocol: TCP
          volumeMounts:
          - name: cephfs
            mountPath: /usr/share/nginx/html   #挂载至容器的路径
      volumes:
      - name: cephfs
        persistentVolumeClaim:
          claimName: cephfs-pvc-nginx      #Pod关联所要绑定的PVC名称

[root@k8s-master1 ceph-cephfs]# kubectl apply -f cephfs-pvc-nginx.yaml 
persistentvolumeclaim/cephfs-pvc-nginx created

[root@k8s-master1 ceph-cephfs]#kubectl apply -f cephfs-deploy-nginx.yaml
deployment.apps/nginx-web created

[root@k8s-master1 ceph-cephfs]# kubectl get pvc -n webapp
[root@k8s-master1 ceph-cephfs]# kubectl get po -n webapp

写入与访问：

[root@k8s-master1 ~]# kubectl get po -n webapp
NAME                           READY   STATUS    RESTARTS   AGE
nginx-web-6898d9dd9b-plwwc   1/1     Running   0          3m57s

[root@k8s-master1 ~]# kubectl exec -it nginx-web-6898d9dd9b-plwwc -n webapp -- /bin/bash -c 'echo hello cephfs-nginx > /usr/share/nginx/html/index.html'

[root@k8s-master1 ~]# kubectl get po -o wide -n webapp | grep nginx-web-6898d9dd9b-plwwc | awk '{print $6}'

[root@k8s-master1 ~]# curl 10.68.159.135
hello cephfs-nginx

删除后重建：

[root@k8s-master1 ~]# cd ceph-cephfs/
[root@k8s-master1 ceph-cephfs]# kubectl delete -f cephfs-deploy-nginx.yaml 
deployment.apps "nginx-web" deleted

[root@k8s-master1 ceph-cephfs]# kubectl get po -n webapp
No resources found in webapp namespace.

[root@k8s-master1 ceph-cephfs]# kubectl get pvc -n webapp

[root@k8s-master1 ceph-cephfs]#kubectl apply -f cephfs-deploy-nginx.yaml
deployment.apps/nginx-web created

[root@k8s-master1 ceph-cephfs]# kubectl get po -n webapp
NAME                           READY   STATUS    RESTARTS   AGE
nginx-web-6898d9dd9b-lhhjf   1/1     Running   0          16s

[root@k8s-master1 ceph-cephfs]# kubectl get po -o wide -n webapp | grep nginx-web-6898d9dd9b-lhhjf | awk '{print $6}'

[root@k8s-master1 ceph-cephfs]# curl 10.68.159.136
hello cephfs-nginx

踩过的点

一是 key 忘了 base64，Pod 起来后挂载报错 Failed to mount，describe 里只显示 secret 无效；二是 monitors 写成单个 mon 地址，那个 mon 一挂全集群的卷都卡住；三是名称空间不一致，secret、PVC、Pod 三者都要在同一个名称空间里，StorageClass 是集群级资源但要在参数里写清 adminSecretNamespace。反复建删测试时加 --rm 跑一次性容器，比留一堆 Completed Pod 干净。

文章版权归作者所有，未经允许请勿转载。

THE END