K8S搭建Nacos集群服务

头像头饰
Awbeci
    阅读 10 分钟
    English

    前言

    最近在学习如何使用k8s搭建nacos服务以及如何使用，记录下来分享给大家。

    准备工作

    • K8S：我使用的是阿里云ACK(阿里的k8s服务)，
    • Nacos：因为我使用的是阿里云RDS的mysql，所以 用的是nacos-group/nacos-k8s的nacos-no-pvc-ingress.yaml文件
      image.png

    开始搭建

    首先我们来查看nacos-no-pvc-ingress.yaml文件

    ###使用自建数据库；使用Ingress发布配置后台###
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  labels:
    app: nacos-headless
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 8848
      name: server
      targetPort: 8848
    - port: 9848
      name: client-rpc
      targetPort: 9848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
      ## 兼容1.4.x版本的选举端口
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  selector:
    app: nacos
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
data:
  mysql.host: "10.127.1.12"
  mysql.db.name: "nacos_devtest"
  mysql.port: "3306"
  mysql.user: "nacos"
  mysql.password: "passwd"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
spec:
  serviceName: nacos-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: k8snacos
          imagePullPolicy: Always
          image: nacos/nacos-server:latest
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
          ports:
            - containerPort: 8848
              name: client
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: NACOS_REPLICAS
              value: "3"
            - name: MYSQL_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.host
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: MODE
              value: "cluster"
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: NACOS_SERVERS
              value: "nacos-0.nacos-headless.default.svc.cluster.local:8848 nacos-1.nacos-headless.default.svc.cluster.local:8848 nacos-2.nacos-headless.default.svc.cluster.local:8848"
  selector:
    matchLabels:
      app: nacos
---
# ------------------- App Ingress ------------------- #
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: default

spec:
  rules:
  - host: nacos-web.nacos-demo.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nacos-headless
          servicePort: server

    接着我们改改上面的配置文件变成我们自己的。

    1. 部署cert-manager
    2. 创建ClusterIssuer
    3. 创建Ingress资源对象
    • 2、ingress配置http 自动跳转到https，使用nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'注解
    • 3、ConfigMap配置自己的Mysql地址和密码
    • 4、StatefulSet集群模式下配置副本数replicas至少为2，否则不起作用
    • 5、StatefulSet设置内存、CPU和模式

    内存、CPU：

    - name: k8snacos
  imagePullPolicy: Always
  image: nacos/nacos-server:latest
  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"

    模式：

    - name: MODE
  # 单机部署，value: "standalone" 
  # 集群部署，value: "cluster"    
  value: "cluster"

    其他的都不用变，如下所示 ：

    #参考：/Users/zhangwei/Development/com.github/nacos-k8s/deploy/nacos/nacos-no-pvc-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    # 添加cert-manager.io/cluster-issuer注解
    cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
    nginx.ingress.kubernetes.io/service-weight: ''
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
spec:
  # 添加ssl证书
  tls:
    - hosts:
        # 替换为您的域名
        - baidu.com        
      secretName: server-seaurl-tls
  rules:
    - host: demo.nacos.com
      http:
        paths:
          - path: /nacos
            backend:
              serviceName: nacos-headless
              servicePort: server
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  labels:
    app: nacos-headless
spec:
  type: ClusterIP
  # headless service
  clusterIP: None
  ports:
    - port: 8848
      name: server
      targetPort: 8848
    - port: 9848
      name: client-rpc
      targetPort: 9848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
    ## 兼容1.4.x版本的选举端口
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  selector:
    app: nacos
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
data:
  mysql.host: "your-aliyun-rds-host"
  mysql.db.name: "nacos"
  mysql.port: "3306"
  mysql.user: "username"
  mysql.password: "password"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
spec:
  serviceName: nacos-headless
  # 单机模式设置成1，集群模式设置成3
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      containers:
        - name: k8snacos
          imagePullPolicy: Always
          image: nacos/nacos-server:latest
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
          ports:
            - containerPort: 8848
              name: client
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: NACOS_REPLICAS
              # 单机模式设置成1，集群模式设置成3
              value: "3"
            - name: MYSQL_SERVICE_HOST # 添加 mysql 访问地址的环境变量
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.host
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: NACOS_APPLICATION_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: NACOS_SERVERS
              # 单机模式设置成nacos-0.xxxx,集群模式要添加：nacos-1.xxxx和nacos-2.xxxxx
              value: "nacos-0.nacos-headless.default.svc.cluster.local:8848 nacos-1.nacos-headless.default.svc.cluster.local:8848 nacos-2.nacos-headless.default.svc.cluster.local:8848"
            - name: MODE
              # 单机部署，value: "standalone" 
              # 集群部署，value: "cluster"    
              value: "cluster"            
  selector:
    matchLabels:
      app: nacos

    然后执行命令来创建Nacos服务：

    kubectl apply -f nacos.yaml

    通过命令查看是否成功

    kubectl get StatefulSet
kubectl get ingress
kubectl get svc
kubectl get pod

    image.png
    从图中可以看出部署nacos服务成功，我们访问试试：
    image.png

    总结

    1、首先我们了解下什么是Service headless，就是type: ClusterIP且clusterIP: None的Service，所以只能通过dns对外去访问你的服务nacos-headless，
    2、单机模式没有使用过，大家可以试试

    注意事项

    1、如果集群模式下只有一个副本会出现问题，至少两个副本

    ------------ 2021-7-5更新-----------------

    部署好之后，我发现本地开发环境启动微服务注册不到ingress的nacos域名：https://demo.nacos.com/nacos。
    原因分析：经过阿里小哥的帮助发现，ingress不需要配置path: /nacos，而直接应该直接使用path: /，可能是因为你加了/nacos，然后k8s去找的时候也加了nacos，变成了https://demo.nacos.com/nacos/nacos(我猜是这样的)，所以最终的ingress应该是：

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nacos-headless
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    # 添加cert-manager.io/cluster-issuer注解
    cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
    nginx.ingress.kubernetes.io/service-weight: ''
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
spec:
  # 添加ssl证书
  tls:
    - hosts:
        # 替换为您的域名
        - baidu.com        
      secretName: server-seaurl-tls
  rules:
    - host: demo.nacos.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nacos-headless
              servicePort: server

    再总结一下：本地开发环境dev，使用域名https://demo.nacos.com来访问，而测试环境test，我们用k8s部署的微服务，如网关等等只能通过k8s dns暴露的service地址来访问，如：http://nacos-headless.default.svc.cluster.local:8848，切记！不同环境下使用的nacos地址不一样！！！

    ------------ 2021-11-12更新-----------------

    因k8s升级到1.22版本，ingress有所调整，如下所示：

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nacos-headless
  # 命名空间，默认：default
  namespace: nacos
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod-http01"
    nginx.ingress.kubernetes.io/service-weight: ''
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
spec:
  tls:
    - hosts:
        - nacos-web.nacos-demo.com        # 替换为您的域名。
      secretName: server-secret-tls
  rules:
    - host: nacos-web.nacos-demo.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nacos-headless
                port:
                  name: server

    引用

    使用cert-manager申请免费的HTTPS证书
    K8S部署Nacos微服务
    k8s部署单节点nacos报错 server is DOWN now, please try again later! 解决
    在 Kubernetes 中使用 DNS 和 Headless Service 发现运行中的 Pod
    K8S容器编排之Headless浅谈

    kubernetesnacos
    Awbeci
    3.1k 声望213 粉丝

    Awbeci

    « 上一篇
    升级SpringCloud、SpringBoot和Alibaba版本
    下一篇 »
    使用Nacos注册和配置SpringCloud微服务

    引用和评论

    推荐阅读
    头像
    SpringCloud-WebClient使用Get和Post请求

    Awbeci阅读 3.7k

    头像
    Nacos服务注册与发现的原理

    无聊的企鹅阅读 313

    头像
    Nacos服务注册与发现

    剣藤犬个阅读 292

    头像
    Nacos源码分析-更新配置时服务端做了什么

    王大锤阅读 100

    头像
    Node.js的Web服务在Nacos中的实践

    维李设论阅读 88

    头像
    Nacos与Eureka的区别有哪些？

    今夜有点儿凉阅读 55

    头像
    JeecgBoot 与分布式事务 Seata v1.7.0 集成实战

    JEECG低代码平台阅读 53

    0 条评论
    得票最新