Delete the namespace that has been stuck in Terminating in KubeSphere

introduce

Been playing with EKS (Elastic Kubernetes Service -- Amazon EKS) and KubeSphere recently. Because EKS and KubeSphere have not been used before, this process is also a trial and error process. When I was using KubeSphere, I found that there is a log service. Driven by curiosity, I created it.

After I created the logging service (KubeSphere Logging System), I found that I didn't want to use it. (Maybe I just want to see what it is.) OCD I want to delete it. So I forcibly deleted him in my EKS:

$ kubectl delete ns kubesphere-logging-system  --force --grace-period=0

The embarrassing thing is that this Namespace was not deleted immediately. I consoled myself, maybe there are other resources that have not been deleted under the Namespace waiting to be deleted, and I will wait. . .

After half an hour, check the deletion progress again:

$ kubectl get  ns  kubesphere-logging-system
NAME                              STATUS        AGE
kubesphere-logging-system         Terminating   6d19h

It looks like the place is stuck in a Terminating state. I tried to find a solution, refer to this isue: http://github.com/kubernetes/kubernetes/issues/60807 . But this method can only be achieved through the API. EKS is hosted in AWS, I have no way to operate the backend of EKS.

Finally I found the answer in this issue: https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-663853215

How to completely delete a namespace

Get namespace details and convert to json

$ kubectl get namespace kubesphere-logging-system -o json > kubesphere-logging-system.json

Open the json file for editing

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-12-31T05:03:58Z",
        "deletionTimestamp": "2022-01-05T08:05:40Z",
        "labels": {
            "kubesphere.io/namespace": "kubesphere-logging-system",
            "kubesphere.io/workspace": "system-workspace"
        },
        "managedFields": [
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            ".": {},
                            "f:kubesphere.io/namespace": {}
                        },
                        "f:ownerReferences": {
                            ".": {},
                            "k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": {
                                ".": {},
                                "f:apiVersion": {},
                                "f:blockOwnerDeletion": {},
                                "f:controller": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:uid": {}
                            }
                        }
                    }
                },
                "manager": "controller-manager",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            "f:kubesphere.io/workspace": {}
                        }
                    },
                    "f:status": {
                        "f:phase": {}
                    }
                },
                "manager": "kubectl",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:status": {
                        "f:conditions": {
                            ".": {},
                            "k:{\"type\":\"NamespaceContentRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            }
                        }
                    }
                },
                "manager": "kube-controller-manager",
                "operation": "Update",
                "time": "2022-01-05T08:05:47Z"
            }
        ],
        "name": "kubesphere-logging-system",
        "ownerReferences": [
            {
                "apiVersion": "tenant.kubesphere.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "Workspace",
                "name": "system-workspace",
                "uid": "6d535470-2592-4f3c-a155-eabc362c339d"
            }
        ],
        "resourceVersion": "7376520",
        "uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e"
    },
    "spec": {
        "finalizers": [
            "kubernetes" # 将此行删除
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

Find the spec and delete kubernetes under finalizers.

details as follows

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-12-31T05:03:58Z",
        "deletionTimestamp": "2022-01-05T08:05:40Z",
        "labels": {
            "kubesphere.io/namespace": "kubesphere-logging-system",
            "kubesphere.io/workspace": "system-workspace"
        },
        "managedFields": [
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            ".": {},
                            "f:kubesphere.io/namespace": {}
                        },
                        "f:ownerReferences": {
                            ".": {},
                            "k:{\"uid\":\"6d535470-2592-4f3c-a155-eabc362c339d\"}": {
                                ".": {},
                                "f:apiVersion": {},
                                "f:blockOwnerDeletion": {},
                                "f:controller": {},
                                "f:kind": {},
                                "f:name": {},
                                "f:uid": {}
                            }
                        }
                    }
                },
                "manager": "controller-manager",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:metadata": {
                        "f:labels": {
                            "f:kubesphere.io/workspace": {}
                        }
                    },
                    "f:status": {
                        "f:phase": {}
                    }
                },
                "manager": "kubectl",
                "operation": "Update",
                "time": "2021-12-31T05:04:01Z"
            },
            {
                "apiVersion": "v1",
                "fieldsType": "FieldsV1",
                "fieldsV1": {
                    "f:status": {
                        "f:conditions": {
                            ".": {},
                            "k:{\"type\":\"NamespaceContentRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionContentFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            },
                            "k:{\"type\":\"NamespaceFinalizersRemaining\"}": {
                                ".": {},
                                "f:lastTransitionTime": {},
                                "f:message": {},
                                "f:reason": {},
                                "f:status": {},
                                "f:type": {}
                            }
                        }
                    }
                },
                "manager": "kube-controller-manager",
                "operation": "Update",
                "time": "2022-01-05T08:05:47Z"
            }
        ],
        "name": "kubesphere-logging-system",
        "ownerReferences": [
            {
                "apiVersion": "tenant.kubesphere.io/v1alpha1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "Workspace",
                "name": "system-workspace",
                "uid": "6d535470-2592-4f3c-a155-eabc362c339d"
            }
        ],
        "resourceVersion": "7376520",
        "uid": "2b76e9b1-75f2-4a2e-a819-73b36aea188e"
    },
    "spec": {
        "finalizers": [
        ]
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All resources successfully discovered",
                "reason": "ResourcesDiscovered",
                "status": "False",
                "type": "NamespaceDeletionDiscoveryFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All legacy kube types successfully parsed",
                "reason": "ParsedGroupVersions",
                "status": "False",
                "type": "NamespaceDeletionGroupVersionParsingFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "All content successfully deleted, may be waiting on finalization",
                "reason": "ContentDeleted",
                "status": "False",
                "type": "NamespaceDeletionContentFailure"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some resources are remaining: fluentbits.logging.kubesphere.io has 1 resource instances",
                "reason": "SomeResourcesRemain",
                "status": "True",
                "type": "NamespaceContentRemaining"
            },
            {
                "lastTransitionTime": "2022-01-05T08:05:47Z",
                "message": "Some content in the namespace has finalizers remaining: fluentbit.logging.kubesphere.io in 1 resource instances",
                "reason": "SomeFinalizersRemain",
                "status": "True",
                "type": "NamespaceFinalizersRemaining"
            }
        ],
        "phase": "Terminating"
    }
}

execute cleanup command

Now we can completely delete this namespace with just one command.

 $ kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json 

After executing, you need to wait for a while and execute the command again to check the Namespace.

 $ kubectl replace --raw "/api/v1/namespaces/kubesphere-logging-system/finalize" -f ./kubesphere-logging-system.json 

final check

$ kubectl get ns kubesphere-logging-system
Error from server (NotFound): namespaces "kubesphere-logging-system" not found

$ kubectl get ns 
NAME                              STATUS   AGE
default                           Active   23d
kubesphere-controls-system        Active   9d
kubesphere-devops-system          Active   9d
kubesphere-devops-worker          Active   16h
kubesphere-monitoring-federated   Active   9d
kubesphere-monitoring-system      Active   9d
kubesphere-sample-dev             Active   8d
kubesphere-system                 Active   9d

When I checked it again, it was no longer there.

This article