A Complete Guide to Migrating Persistent Volumes Across StorageClasses in Kubernetes

Hi, my name is Michelangelo.

KubeSphere 3.3.0 (no accident~) is going to be GA this week. As a KubeSphere fan, I can't wait to install the RC version and try it out. I open all the components in one operation, and I find it a little embarrassing after the installation. : I'm using persistent storage wrong.

There are two storage classes (StorageClass) in my K8s cluster, one is the local storage provided by OpenEBS, the other is the distributed storage provided by QingCloud CSI , and the default StorageClass is the local-hostpath provided by OpenEBS, so the state of KubeSphere Components by default use local storage to save data.

Mistake, I originally wanted to use distributed storage as the default storage, but I forgot to set csi-qingcloud as the default StorageClass. Anyway, I made a mistake. Although reloading can solve 99% of the problems, as a mature YAML engineer, reloading is impossible, and this problem must be solved without reloading to reflect my temperament!

In fact, I am not the only one who has encountered this situation. Many people will find that StorageClass is used wrong after installing a whole set of products in a confused way. At this time, it may not be so easy to change it back. This is not a coincidence, this is not the case, this article is to help you solve this problem.

ideas

Let's first think about what needs to be done to change the StorageClass. First, you need to reduce the number of copies of the application to 0, then create a new PVC, copy the data of the old PV to the new PV, then let the application use the new PV, expand the copy to the original number, and finally copy the old PV delete. During this whole process, it is also necessary to prevent Kubernetes from deleting the PV when the PVC is deleted.

Of course, some CSI drivers or storage backends may have more convenient data migration techniques, but this article provides a more general solution, regardless of the backend storage.

The persistent volume declaration (PVC) used by KubeSphere 3.3.0 after enabling all components is as follows:

This article takes Elasticsearch as an example to demonstrate how to replace Elasticsearch's storage from local storage to distributed storage.

Backup PVCs and PVs

The first step is to back up the PVC and PV. If the subsequent operation fails, there is still room for repentance.

 $ kubectl -n kubesphere-logging-system get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
data-elasticsearch-logging-data-0        Bound    pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9   20Gi       RWO            local-hostpath   28h
data-elasticsearch-logging-data-1        Bound    pvc-0851350a-270e-4d4d-af8d-081132c1775b   20Gi       RWO            local-hostpath   28h
data-elasticsearch-logging-discovery-0   Bound    pvc-8f32fc97-3d6e-471a-8121-655991d945a8   4Gi        RWO            local-hostpath   28h

$ kubectl -n kubesphere-logging-system get pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 -o yaml > pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9.yaml
$ kubectl -n kubesphere-logging-system get pv pvc-0851350a-270e-4d4d-af8d-081132c1775b -o yaml > pvc-0851350a-270e-4d4d-af8d-081132c1775b.yaml

$ kubectl -n kubesphere-logging-system get pvc data-elasticsearch-logging-data-0 -o yaml > data-elasticsearch-logging-data-0.yaml
$ kubectl -n kubesphere-logging-system get pvc data-elasticsearch-logging-data-1 -o yaml > data-elasticsearch-logging-data-1.yaml

copy data

Regardless of whether the PV's accessModes are ReadWriteOnce or ReadWriteMany , the number of copies of the application must be reduced to 0 before copying data, because ReadWriteOne mode only allows one Pod to be mounted at the same time, and new Pods cannot be mounted, and ReadWriteMany mode If the number of copies is not reduced If it is 0, new data may be written while copying data. So in any case, reduce the number of copies to 0.

 $ kubectl -n kubesphere-logging-system get sts
NAME                              READY   AGE
elasticsearch-logging-data        2/2     28h
elasticsearch-logging-discovery   1/1     28h

$ kubectl -n kubesphere-logging-system scale sts elasticsearch-logging-data --replicas=0

$ kubectl -n kubesphere-logging-system get sts
NAME                              READY   AGE
elasticsearch-logging-data        0/0     28h
elasticsearch-logging-discovery   1/1     28h

Create a new PVC called new-data-elasticsearch-logging-data-0 with the same capacity as data-elasticsearch-logging-data-0 and specify storageClassName as the new StorageClass.

Create a Deployment, mount both the new PV and the old PV into it, and then copy the data of the old PV to the new PV.

Click "Create" on the "Workload" interface, and paste the following YAML into it.

 apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kubesphere-logging-system
  labels:
    app: datacopy
  name: datacopy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: datacopy
  template:
    metadata:
      labels:
        app: datacopy
    spec:
      containers:
        - name: datacopy
          image: ubuntu
          command:
            - 'sleep'
          args:
            - infinity
          volumeMounts:
            - name: old-pv
              readOnly: false
              mountPath: /mnt/old
            - name: new-pv
              readOnly: false
              mountPath: /mnt/new
      volumes:
        - name: old-pv
          persistentVolumeClaim:
            claimName: data-elasticsearch-logging-data-0
        - name: new-pv
          persistentVolumeClaim:
            claimName: new-data-elasticsearch-logging-data-0

This Deployment mounts both the new PV and the old PV, and later we will copy the data from the old PV to the new PV.

After the Pod starts successfully, click the terminal icon of the container to enter the terminal of the container.

In the container, first verify whether the mount point of the old PV contains application data, and whether the mount point of the new PV is empty, and then execute the command (cd /mnt/old; tar -cf - .) | (cd /mnt/new; tar -xpf -) to ensure that the ownership and permissions of all data are inherited.

After execution, verify that the new PV's mount point contains the old PV's data, and that ownership and permissions are properly inherited.

The task of copying data here is complete, now we need to reduce the number of copies of datacopy to 0.

Migrating PVCs

The ideal state for migrating storage is to use the old PVC and point it to the new PV so that the workload's YAML configuration manifest does not require any changes. But the binding relationship between PVC and PV cannot be changed. To unbind them, you must delete the old PVC first, create a PVC with the same name, and bind the old PV to it.

It should be noted that the default PV recycling policy is Delete . Once a PVC is deleted, the PV bound to it and the data in the PV will be deleted. This is something we don't want to see, so we need to modify the recycling strategy so that the PV can be preserved when the PVC is deleted.

In fact , the global recycling policy (reclaimPolicy) can be set through StorageClass . If not set, the default is Delete . You can use the command kubectl describe pv <pv-name> to view the PV reclaim policy (reclaimPolicy):

 $ kubectl describe pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9
Name:              pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9
Labels:            openebs.io/cas-type=local-hostpath
Annotations:       pv.kubernetes.io/provisioned-by: openebs.io/local
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-hostpath
Status:            Bound
Claim:             kubesphere-logging-system/data-elasticsearch-logging-data-0
Reclaim Policy:    Delete
...

$ kubectl describe pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Name:              pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.qingcloud.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-qingcloud-com]
StorageClass:      csi-qingcloud
Status:            Bound
Claim:             kubesphere-logging-system/new-data-elasticsearch-logging-data-0
Reclaim Policy:    Delete
...

We can set the recycling policy of the old and new PVs to Retain through the patch command.

 $ kubectl patch pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 patched

$ kubectl patch pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9 patched

⚠️Note: This command has no impact on the stability and availability of PV and can be executed at any time.

Both new and old PVCs can now be deleted without any impact on the PVs.

 $ kubectl -n kubesphere-logging-system delete pvc data-elasticsearch-logging-data-0 new-data-elasticsearch-logging-data-0
persistentvolumeclaim "data-elasticsearch-logging-data-0" deleted
persistentvolumeclaim "new-data-elasticsearch-logging-data-0" deleted

Before creating the final PVC, we must ensure that the newly created PVC can be bound to the new PV. You can see that the new PV is currently in a released state and cannot be bound by the new PVC through the following command:

 $ kubectl describe pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Name:              pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.qingcloud.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-qingcloud-com]
StorageClass:      csi-qingcloud
Status:            Released
Claim:             kubesphere-logging-system/new-data-elasticsearch-logging-data-0
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          20Gi
...

This is because the PV still references the deleted PVC in spec.claimRef :

 $ kubectl get pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  ...
  name: pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
  ...
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: new-data-elasticsearch-logging-data-0
    namespace: kubesphere-logging-system
    resourceVersion: "657019"
    uid: f4e96f69-b3be-4afe-bb52-1e8e728ca55e
  ...
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-qingcloud
  volumeMode: Filesystem

In order to solve this problem, you can directly edit the PV through the command kubectl edit pv <pv-name> , and delete all the contents of claimRef . Then check that the PV is already available (Available):

 $ kubectl describe pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Name:              pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: disk.csi.qingcloud.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/disk-csi-qingcloud-com]
StorageClass:      csi-qingcloud
Status:            Available
Claim:
Reclaim Policy:    Retain
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          20Gi

Ultimately we need to create a new PVC with the same name as the old PVC, with the same parameters as possible as possible:

• The name of the new PVC is the same as the name of the old PVC;
• spec.volumeName point to the new PV;
• The new PVC's metadata.annotations and metadata.labels are kept the same as the old PVC, as these values may affect application deployment (eg Helm chart, etc.).

The final PVC contents are as follows:

 apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: elasticsearch
    component: data
    release: elasticsearch-logging
    role: data
  name: data-elasticsearch-logging-data-0
  namespace: kubesphere-logging-system
spec:
  storageClassName: csi-qingcloud
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  volumeMode: Filesystem
  volumeName: pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e

Click "Create" on the "Volume Declaration" page:

Select "Edit YAML", copy and paste the above YAML content, and then click "Create":

Finally, you can see that the new PVC and PV are all in Bound state:

 $ kubectl -n kubesphere-logging-system get pvc data-elasticsearch-logging-data-0
NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-elasticsearch-logging-data-0   Bound    pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            csi-qingcloud   64s

$ kubectl get pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                         STORAGECLASS    REASON   AGE
pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            Retain           Bound    kubesphere-logging-system/data-elasticsearch-logging-data-0   csi-qingcloud            11h

do it again

So far, we have only migrated the data of data-elasticsearch-logging-data-0 05d7b7e5a3b951525bf4b227e7220a55---, for data-elasticsearch-logging-data-1 , just repeat the above steps, remember to change the PVC in datacopy to data-elasticsearch-logging-data-1 new-data-elasticsearch-logging-data-0 , and the configuration content in other places should also be changed to the new one.

restore workload

All storage is now migrated, the PVC name remains the same, and the PV uses the new storage.

 $ kubectl get pv -A|grep elasticsearch-logging-data
pvc-0851350a-270e-4d4d-af8d-081132c1775b   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-1        local-hostpath            40h
pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-0        local-hostpath            40h
pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563   20Gi       RWO            Retain           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-1        csi-qingcloud             9m53s
pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            Retain           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-0        csi-qingcloud             11h

$ kubectl -n kubesphere-logging-system get pvc|grep elasticsearch-logging-data
data-elasticsearch-logging-data-0        Bound    pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            csi-qingcloud    27m
data-elasticsearch-logging-data-1        Bound    pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563   20Gi       RWO            csi-qingcloud    3m49s

Restore the workload's replicas to the previous number:

 $ kubectl -n kubesphere-logging-system scale sts elasticsearch-logging-data --replicas=2
statefulset.apps/elasticsearch-logging-data scaled

$ kubectl -n kubesphere-logging-system get pod -l app=elasticsearch,component=data
NAME                           READY   STATUS    RESTARTS   AGE
elasticsearch-logging-data-0   1/1     Running   0          4m12s
elasticsearch-logging-data-1   1/1     Running   0          3m42s

Perfect!

One last bit of finishing touches, we need to reset the reclamation policy for all new PVs to Delete :

 $ kubectl patch pv pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563 -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
persistentvolume/pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563 patched

$ kubectl patch pv pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
persistentvolume/pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e patched

$ kubectl get pv -A|grep elasticsearch-logging-data
pvc-0851350a-270e-4d4d-af8d-081132c1775b   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-1        local-hostpath            40h
pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9   20Gi       RWO            Retain           Released   kubesphere-logging-system/data-elasticsearch-logging-data-0        local-hostpath            40h
pvc-d0acd2e7-ee1d-47cf-8506-69147fe25563   20Gi       RWO            Delete           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-1        csi-qingcloud             15m
pvc-f4e96f69-b3be-4afe-bb52-1e8e728ca55e   20Gi       RWO            Delete           Bound      kubesphere-logging-system/data-elasticsearch-logging-data-0        csi-qingcloud             11h

Finally, you can delete all the old PVs:

 $ kubectl delete pv pvc-0851350a-270e-4d4d-af8d-081132c1775b
persistentvolume "pvc-0851350a-270e-4d4d-af8d-081132c1775b" deleted

$ kubectl delete pv pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9
persistentvolume "pvc-9aed3d1b-09a6-4fe3-8adc-9195a2bbb2b9" deleted

simpler solution

Although the above solution perfectly solves the problem, the steps are more complicated. Is there a more concise method?

You can try the cloud-native backup disaster recovery SaaS service launched by Qingyun. You can easily complete the free migration of data in a multi-cloud heterogeneous environment without deploying and maintaining a local backup infrastructure, so as to achieve multi-location and on-demand data protection and application. High availability. Moreover, the price is relatively close to the people, and it is friendly to the prostitutes. It provides 100GB of free storage , and it is enough to migrate a few PVs.

It is very simple to use, first register an account , and then import the Kubernetes cluster. If you choose to connect to the Kubernetes cluster through a proxy, you need to execute the command in the red box to install the proxy in the Kubernetes cluster.

Then create a new managed repository.

Next, create a backup plan directly and select Direct Copy .

After a successful backup, delete the PVCs and PVs in the cluster and reduce the number of replicas for the workload to 0. Finally, create a recovery plan. Note that the source storage type name is local-hostpath and the target storage type name is set to the storage you want to migrate, so that the recovered PV uses the new StorageClass.

Done.

Summarize

This article describes how to migrate data from an existing PV in a Kubernetes cluster to a new PV, and create a PVC with the same name to point to the new PV, thus completing the data migration of the application without any changes to the application's configuration manifest. Finally, it introduces how to simplify the migration process through cloud-native backup disaster recovery SaaS services.

This article is published by OpenWrite , a multi-post blog platform!