K8S builds Nacos cluster service

Preface

Recently I am learning how to use k8s to build nacos service and how to use it, record it and share it with everyone.

Ready to work

• K8S: I am using Alibaba Cloud ACK (Ali’s k8s service),
• Nacos: Because I use mysql of Cloud RDS, I use the nacos-no-pvc-ingress.yaml file of 1618e758cdb2a9 nacos-group/nacos-k8s
  

Start building

First, let's check the nacos-no-pvc-ingress.yaml file

###使用自建数据库；使用Ingress发布配置后台###
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  labels:
    app: nacos-headless
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 8848
      name: server
      targetPort: 8848
    - port: 9848
      name: client-rpc
      targetPort: 9848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
      ## 兼容1.4.x版本的选举端口
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  selector:
    app: nacos
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
data:
  mysql.host: "10.127.1.12"
  mysql.db.name: "nacos_devtest"
  mysql.port: "3306"
  mysql.user: "nacos"
  mysql.password: "passwd"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
spec:
  serviceName: nacos-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: k8snacos
          imagePullPolicy: Always
          image: nacos/nacos-server:latest
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
          ports:
            - containerPort: 8848
              name: client
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: NACOS_REPLICAS
              value: "3"
            - name: MYSQL_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.host
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: MODE
              value: "cluster"
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: NACOS_SERVERS
              value: "nacos-0.nacos-headless.default.svc.cluster.local:8848 nacos-1.nacos-headless.default.svc.cluster.local:8848 nacos-2.nacos-headless.default.svc.cluster.local:8848"
  selector:
    matchLabels:
      app: nacos
---
# ------------------- App Ingress ------------------- #
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: default

spec:
  rules:
  - host: nacos-web.nacos-demo.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nacos-headless
          servicePort: server

Then we changed the above configuration file to become our own.

• supports https. Here is an article on 1618e758cdb3a0 using cert-manager to apply for a free HTTPS certificate detailed process is not described. The process is:

1. Deploy cert-manager
2. Create ClusterIssuer
3. Create Ingress resource object

• nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' configures http to automatically jump to https, using 0618e758cdb41c annotation
• 3. Configure your own Mysql address and password in ConfigMap
• 4. The number of replicas configured in the StatefulSet cluster mode is at least 2, otherwise it will not work
• 5. StatefulSet sets memory, CPU and mode

Memory, CPU:

- name: k8snacos
  imagePullPolicy: Always
  image: nacos/nacos-server:latest
  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"

model:

- name: MODE
  # 单机部署，value: "standalone" 
  # 集群部署，value: "cluster"    
  value: "cluster"  

Nothing else needs to be changed, as shown below:

#参考：/Users/zhangwei/Development/com.github/nacos-k8s/deploy/nacos/nacos-no-pvc-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    # 添加cert-manager.io/cluster-issuer注解
    cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
    nginx.ingress.kubernetes.io/service-weight: ''
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
spec:
  # 添加ssl证书
  tls:
    - hosts:
        # 替换为您的域名
        - baidu.com        
      secretName: server-seaurl-tls
  rules:
    - host: demo.nacos.com
      http:
        paths:
          - path: /nacos
            backend:
              serviceName: nacos-headless
              servicePort: server
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  labels:
    app: nacos-headless
spec:
  type: ClusterIP
  # headless service
  clusterIP: None
  ports:
    - port: 8848
      name: server
      targetPort: 8848
    - port: 9848
      name: client-rpc
      targetPort: 9848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
    ## 兼容1.4.x版本的选举端口
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  selector:
    app: nacos
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
data:
  mysql.host: "your-aliyun-rds-host"
  mysql.db.name: "nacos"
  mysql.port: "3306"
  mysql.user: "username"
  mysql.password: "password"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
spec:
  serviceName: nacos-headless
  # 单机模式设置成1，集群模式设置成3
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: k8snacos
          imagePullPolicy: Always
          image: nacos/nacos-server:latest
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
          ports:
            - containerPort: 8848
              name: client
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: NACOS_REPLICAS
              # 单机模式设置成1，集群模式设置成3
              value: "3"
            - name: MYSQL_SERVICE_HOST # 添加 mysql 访问地址的环境变量
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.host
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: NACOS_APPLICATION_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: NACOS_SERVERS
              # 单机模式设置成nacos-0.xxxx,集群模式要添加：nacos-1.xxxx和nacos-2.xxxxx
              value: "nacos-0.nacos-headless.default.svc.cluster.local:8848 nacos-1.nacos-headless.default.svc.cluster.local:8848 nacos-2.nacos-headless.default.svc.cluster.local:8848"
            - name: MODE
              # 单机部署，value: "standalone" 
              # 集群部署，value: "cluster"    
              value: "cluster"            
  selector:
    matchLabels:
      app: nacos

Then execute the command to create the Nacos service:

kubectl apply -f nacos.yaml

Check whether it is successful through the command

kubectl get StatefulSet
kubectl get ingress
kubectl get svc
kubectl get pod

It can be seen from the figure that the deployment of the nacos service is successful, let's try it out:

Summarize

1. First of all, we understand what Service headless is, that is, Service with type: ClusterIP and clusterIP: None, so you can only access your service nacos-headless through dns.
2. The stand-alone mode has not been used before, you can try

Precautions

1. If there is only one copy in the cluster mode, there will be problems, at least two copies

------------ 2021-7-5 update-----------------

After the deployment, I found that the local development environment started the microservice to register the ingress nacos domain name: https://demo.nacos.com/nacos .
reason for with the help of Ali brother found that ingress does not need to configure path: /nacos, and should directly use path: / . It may be because you added /nacos and then when k8s went to find it. After nacos, it became https://demo.nacos.com/nacos/nacos (I guess so), so the final ingress should be:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    # 添加cert-manager.io/cluster-issuer注解
    cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
    nginx.ingress.kubernetes.io/service-weight: ''
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
spec:
  # 添加ssl证书
  tls:
    - hosts:
        # 替换为您的域名
        - baidu.com        
      secretName: server-seaurl-tls
  rules:
    - host: demo.nacos.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nacos-headless
              servicePort: server

To summarize: local development environment dev, use the domain name https://demo.nacos.com to access, and the test environment test, we use k8s deployment of microservices, such as gateways, etc. can only be accessed through the service address exposed by k8s dns, such as: http://nacos-headless.default.svc.cluster.local:8848 , Remember! The nacos address used in different environments is different! ! !

------------ 2021-11-12 update-----------------

Due to the upgrade of k8s to version 1.22, ingress has been adjusted, as shown below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nacos-headless
  # 命名空间，默认：default
  namespace: nacos
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
    nginx.ingress.kubernetes.io/service-weight: ''
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
spec:
  tls:
    - hosts:
        - nacos-web.nacos-demo.com        # 替换为您的域名。
      secretName: server-secret-tls
  rules:
    - host: nacos-web.nacos-demo.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nacos-headless
                port:
                  name: server

Quote

Use cert-manager to apply for a free HTTPS certificate
K8S deploys Nacos
k8s deployment single node nacos report error server is DOWN now, please try again later! Solve
Use DNS and Headless Service in Kubernetes to discover the running Pod
Headless Talk on