Embrace the blue-green deployment and promote the smooth upgrade of the AKS cluster version

Azure Kubernetes (AKS) is a Kubernetes cluster hosted on Microsoft Azure cloud. It can be used to quickly deploy Kubernetes clusters, combined with other Azure services and functions, to simplify daily operation and maintenance, and easily achieve business application flexibility.

Presumably, many small partners are already using AKS services extensively. However, in use, it is inevitable to find that the current Kubernetes community is booming and version updates are more frequent. This is actually a good thing, it can fix various bugs that have been found more quickly, and at the same time, it can also bring us various innovative functions through upgrades.

However, how to upgrade the deployed AKS cluster to the latest version in order to obtain various new functions?

For the upgrade of the AKS managed version, you can upgrade the AKS cluster in-situ with one click, or upgrade the working nodes in the cluster one by one. For small-scale or non-critical business clusters, it is very convenient to use the hosting upgrade function of AKS; however, for large-scale clusters or business-critical applications, the above-mentioned one-click in-situ upgrade method has a long overall upgrade time period. There may be a failure, and if a failure occurs, it will not be able to roll back and other limitations.

So is there any method that is simple to operate and has good results, and can take into account the advantages of the above two methods at the same time?

Blue-green deployment

Small partners who have been engaged in application development or operation and maintenance should be familiar with this concept. This application publishing model can gradually transfer user traffic from the previous version of the application to the almost identical new version (the old version can be called the blue environment, and the new version can be called the green environment). Once the production flow is completely transferred from the blue to the green environment, the blue environment can be changed to standby in case of an accident, which can be rolled back and restored. After a period of observation is stable, it can be completely cleaned up to save resources.

In fact, for the AKS cluster upgrade, we can also adopt a similar idea: create a new AKS cluster, and use the blue-green deployment method to switch and upgrade. In this way, the second-level switching can be realized, and the backup cluster can be rolled back at any time, which is faster and safer.

However, the architecture and operation of the solution are slightly complicated. This article will lead you to gradually build such a set of switching and upgrading architecture solutions. In the following example, we will use the classic web application scenario, but it can also be extended to other AKS cluster application scenarios.

Architecture overview

The above figure shows a classic Web application architecture diagram. We have selected the most concise resources to make the presentation and description as concise as possible.

Three subnets are divided into a virtual network. The first subnet is placed with application gateways as a load balancer for external services; the latter two subnets usually have only one subnet with one AKS cluster, and business systems are deployed. The other subnet is used to create a new AKS cluster during version upgrade. The AKS cluster uses advanced network CNI to simplify the network mode and facilitate communication with the application gateway.

Pod Identity is deployed in the AKS cluster, and the Pod in AKS can be authorized to manage the application gateway through the Pod Identity of Azure AD.

The full name of AGIC is Application Gateway Ingress Controller. In addition to distributing network traffic from the application gateway to the corresponding Pod, it also monitors changes in some Kubernetes resources. It can automatically update the back-end pool of the application gateway when AKS is scaled, and in AKS When the cluster is switched, the back-end pool of the application gateway is updated synchronously. Using AGIC to dynamically update the back-end of the application gateway is our core idea for blue-green deployment.

Resource deployment

Deploy basic resources such as the network

First, you need to use CLI to quickly create the basic resources of the current environment.

Create a resource group:

AZ_REGION=ChinaNorth2
RESOURCE_GROUP=AKS_Upgrade
az group create -n $RESOURCE_GROUP -l $AZ_REGION

Create VNET and subnet:

VNET_NAME=AksVnet
APPGW_SUBNET=AppGwSubnet
AKS_SUBNET=AksSubnet
 
network  vnet  create on $VNET_NAME \
-g $RESOURCE_GROUP \
-l $AZ_REGION \
--address-prefix 10.0.0.0/8 \
--subnet-name $APPGW_SUBNET --subnet-prefix 10.1.0.0/16
 
az network vnet subnet create \
-g $RESOURCE_GROUP \
-n $AKS_SUBNET \
--address-prefixes 10.240.0.0/16 \
--vnet-name $VNET_NAME

Deploy the Azure resources of the current AKS version

Create a public IP:

APPGW_IP=AppGatewayIp
az network public-ip create -n $APPGW_IP \
-g $RESOURCE_GROUP \
--allocation-method Static \
--sku Standard

Create an application gateway:

APP_GATEWAY=AppGateway
az network application-gateway create -n $APP_GATEWAY \
 -g $RESOURCE_GROUP \
 -l $AZ_REGION \
 --vnet-name $VNET_NAME \
 --subnet $APPGW_SUBNET \
 --sku Standard_v2 \
 --public-ip-address $APPGW_IP

Create the old AKS cluster. Use the current default mainstream AKS version. First, get the subnet ID of the AKS cluster created earlier.

AKS_SUBNET_ID=$(az network vnet subnet show -g $RESOURCE_GROUP --vnet-name $VNET_NAME --name $AKS_SUBNET --query id -o tsv)

Create an AKS cluster (at the time of writing, the mainstream AKS version is 1.19.11):

AKS_OLD=old
az aks create -n $AKS_OLD \
-g $RESOURCE_GROUP \
-l $AZ_REGION \
--generate-ssh-keys \
--network-plugin azure \
--enable-managed-identity \
--vnet-subnet-id $AKS_SUBNET_ID

Integrate the application gateway with the current version of AKS

We will use the Azure service principal to authorize the AKS cluster to manage the configuration of the application gateway.

First connect to the AKS cluster:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_OLD

Then you can use the familiar kubectl to manage the AKS cluster.

Install Helm and run the following command to add the application-gateway-kubernetes-ingress Helm package. Our AKS cluster has Kubernetes RBAC enabled, so the following command can be used:

kubectl create serviceaccount --namespace kube-system tiller-sa
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install aad-pod-identity aad-pod-identity/aad-pod-identity

The above command will return the following result:

NAME: aad-pod-identity
LAST DEPLOYED: Tue Jun 29 08:14:30 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed AAD Pod Identity in your Kubernetes cluster!
…

After waiting a minute or two, run the following command:

kubectl get po -o wide
NAME                                    READY   STATUS    RESTARTS   AGE   IP            NODE                                NOMINATED NODE   READINESS GATES
aad-pod-identity-mic-787c5958fd-kmx9b   1/1     Running   0          71s   10.240.0.33   aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-mic-787c5958fd-nkpv4   1/1     Running   0          72s   10.240.0.63   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-mhp86              1/1     Running   0          72s   10.240.0.4    aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-nmi-sjpvw              1/1     Running   0          72s   10.240.0.35   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-xnfxh              1/1     Running   0          72s   10.240.0.66   aks-nodepool1-94448771-vmss000002   <none>           <none>

As you can see, several related Pods are already running.

Then use Helm to install Application Gateway Ingress Controller:

helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

Copy the following YAML file and save it as helm_agic.yaml to configure AGIC:

# This file contains the essential configs for the ingress controller helm chart
# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3
 
################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
    subscriptionId: <subscriptionId>
    resourceGroup: <resourceGroupName>
    name: <applicationGatewayName>
    environment: AzureChinaCloud
 
    # Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false
 
################################################################################
# Specify which kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all acessible namespaces.
#
# kubernetes:
#   watchNamespace: <namespace>
 
################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
# armAuth:
#    type: aadPodIdentity
#    identityResourceID: <identityResourceId>
#    identityClientID:  <identityClientId>
 
## Alternatively you can use Service Principal credentials
armAuth:
    type: servicePrincipal
    secretJSON: <<Generate value with: "az ad sp create-for-rbac --sdk-auth | base64 -w0">>
 
################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
    enabled: true # true/false

We fill in the parameter values in the above configuration file one by one.

• <subscriptionId>: Obtained through az account show --query id -o tsv;
• <resourceGroupName>: The value takes the $RESOURCE_GROUP environment variable;
• <applicationGatewayName>: The value takes the $APP_GATEWAY environment variable.

The value of secretJSON is obtained using the az ad sp create-for-rbac --sdk-auth | base64 -w0 command, which is a long base64-encoded string of more than 800 bytes.

Because our AKS cluster has RBAC enabled, the last configuration rbac is set to true.

Finally, execute the following command to install:

helm install agic application-gateway-kubernetes-ingress/ingress-azure -f helm_agic.yaml

The above command will return the following result:

W0629 08:16:47.733467   16087 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
NAME: agic
LAST DEPLOYED: Tue Jun 29 08:16:48 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing ingress-azure:1.4.0.

Your release is named agic.
The controller is deployed in deployment agic-ingress-azure.

Configuration Details:
----------------------
 * AzureRM Authentication Method:
    - Use AAD-Pod-Identity
 * Application Gateway:
    - Subscription ID : 3d07553f-f6a8-455f-9de6-876fbcc00bb4
    - Resource Group  : AKS_Upgrade
    - Application Gateway Name : AppGateway
 * Kubernetes Ingress Controller:
    - Watching All Namespaces
    - Verbosity level: 3

Then run the following commands:

kubectl get po -o wide
NAME                                    READY   STATUS    RESTARTS   AGE     IP            NODE                                NOMINATED NODE   READINESS GATES
aad-pod-identity-mic-787c5958fd-kmx9b   1/1     Running   0          4m54s   10.240.0.33   aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-mic-787c5958fd-nkpv4   1/1     Running   0          4m55s   10.240.0.63   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-mhp86              1/1     Running   0          4m55s   10.240.0.4    aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-nmi-sjpvw              1/1     Running   0          4m55s   10.240.0.35   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-xnfxh              1/1     Running   0          4m55s   10.240.0.66   aks-nodepool1-94448771-vmss000002   <none>           <none>
agic-ingress-azure-8d9d85dd9-z8dwh      1/1     Running   0          2m37s   10.240.0.70   aks-nodepool1-94448771-vmss000002   <none>           <none>

It is found that the newly-built agic-ingress-azure Pod is also running normally.

---

So far, we have successfully deployed related resources and achieved the integration of application gateway and AKS. In the next article, three tasks will be involved: application deployment, AKS new cluster deployment, and AKS version switching. Stay tuned!