使用ceph-rbd 作为k8s后端存储

前两篇说了ceph集群搭建及rbd的使用等一些功能，然而ceph毕竟是作为一个存储，需要集成到应用中，rdb支持多种设备使用，大体上有libvirt,OpenStack,CloudStack,Kubernetes现在使用ceph作为k8s的存储，也就是ceph-rbd和StorageClass相结合，为pod提供存储，没有k8s集群可以使用kubeadm搭建

Ceph集群中创建资源等配置

创建存储池并设定pg和pg_num

ceph osd pool create kubernetes 32 32

初始化存储池

rbd pool init kubernetes

创建client用户访问进程池

ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'

Kubernetes安装ceph-rbd驱动配置

每台k8s节点安装ceph-rbd命令包

yum install ceph-common

设置ceph-cm配置文件

• clusterID为你ceph集群id
• monitors为你ceph的mon服务地址

---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "9bb7af93-99f3-4254-b0b3-864f5d1413e4",
        "monitors": [
          "192.168.21.100:6789",
          "192.168.21.101:6789",
          "192.168.21.102:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
  namespace: ceph-st
kubectl apply -f csi-config-map.yaml

创建访问ceph集群的证书，userID为上面创建的name，userKey为生成的key，忘记使用ceph auth list查看

cat <<EOF > csi-rbd-secret.yaml
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: AQCb7mpfCr4oGhAAalv/q8WSnUE/vyu59Ge3Hg==
EOF

安装ceph-rbd驱动程序

wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

[root@k8s-master rbd]# cat ceph-csi-encryption-kms-config.yaml 
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    {
      "vault-test": {
        "encryptionKMSType": "vault",
        "vaultAddress": "http://vault.default.svc.cluster.local:8200",
        "vaultAuthPath": "/v1/auth/kubernetes/login",
        "vaultRole": "csi-kubernetes",
        "vaultPassphraseRoot": "/v1/secret",
        "vaultPassphrasePath": "ceph-csi/",
        "vaultCAVerify": "false"
      }
    }
metadata:
  name: ceph-csi-encryption-kms-config
  namespace: ceph-st
kubectl apply -f ./

创建StorageClass

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
   namespace: ceph-st
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 9bb7af93-99f3-4254-b0b3-864f5d1413e4
   pool: kubernetes
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: ceph-st
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: ceph-st
   imageFeatures: "layering"
reclaimPolicy: Delete
mountOptions:
   - discard

创建PVC测试功能

查看服务状态

创建文件设备

[root@k8s-master rbd]# cat pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem #指定使用文件格式
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
---
apiVersion: v1
kind: Pod
metadata:
  name: csi-rbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

查看各个组件是否正常

• k8s服务

[root@k8s-master rbd]# kubectl get pvc | grep rbd-pvc
rbd-pvc                                       Bound    pvc-6c63b1c3-0b81-4a51-8c1b-e3d0acae1b6c   1Gi        RWO            csi-rbd-sc           22d
[root@k8s-master rbd]# kubectl get pv | grep rbd-pvc
pvc-6c63b1c3-0b81-4a51-8c1b-e3d0acae1b6c   1Gi        RWO            Delete           Bound      default/rbd-pvc

• ceph服务

[root@node-1 ~]# rbd -p kubernetes ls
csi-vol-4e6663e7-fd80-11ea-9f64-c25f47044f33
csi-vol-a2195114-fd7e-11ea-9f64-c25f47044f33

创建块设备pvc这样挂载的是一个块文件

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
  
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
    - name: fc-container
      image: fedora:26
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc