Deploy RabbitMQ on K8s cluster in practice

Author: Old Z, a cloud native enthusiast, currently focusing on cloud native operation and maintenance, KubeSphere Ambassador.

foreword

Knowledge point

Rating: entry level
RabbitMQ single node installation and deployment
RabbitMQ cluster installation and deployment
GitOps operation and maintenance ideas

Demo server configuration

CPU name	IP	CPU	Memory	system disk	data disk	use
zdeops-master	192.168.9.9	2	4	40	200	Ansible operation and maintenance control node
ks-k8s-master-0	192.168.9.91	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-1	192.168.9.92	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
ks-k8s-master-2	192.168.9.93	4	16	40	200+200	KubeSphere/k8s-master/k8s-worker/Ceph
storage-node-0	192.168.9.95	2	8	40	200+200	ElasticSearch/GlusterFS
storage-node-0	192.168.9.96	2	8	40	200+200	ElasticSearch/GlusterFS
storage-node-0	192.168.9.97	2	8	40	200+200	ElasticSearch/GlusterFS
harbor	192.168.9.89	2	8	40	200	Harbor
total	8	twenty two	84	320	2800

The demo environment involves software version information

OS: CentOS-7.9-x86_64
Ansible: 2.8.20
KubeSphere: 3.3.0
Kubernetes: v1.24.1
Rook: v1.9.7
Ceph: v16.2.9
GlusterFS: 9.5.1
ElasticSearch: 7.17.5
Harbor: 2.5.1
RabbitMQ: 3.9.22
RabbitMQ Cluster Operator: 1.14.0

Introduction

How to deploy RabbitMQ single node on K8s cluster? How is RabbitMQ cluster deployed on K8s cluster? 60 minutes to take you into the actual combat.

Single node RabbitMQ deployment

Sorting out ideas

StatefulSet
Headless Service: For internal services
External Service: For external management
Secrets: Manage usernames and passwords
Image: rabbitmq:3.9.22-management (with management plugin) officially provided by DockerHub

Prepare for offline mirroring

This process is optional, and the offline intranet environment is available. If the intranet image is not configured, pay attention to changing the image of the container to the default value in the subsequent resource configuration list.

Perform the following operations on a server that can access both the Internet and the intranet Harbor warehouse.

Download mirror

 docker pull rabbitmq:3.9.22-management

re-tag

 docker tag rabbitmq:3.9.22-management registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

Push to private mirror repository

 docker push registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

clean up temporary images

 docker rmi rabbitmq:3.9.22-management
docker rmi registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

Resource configuration checklist

rabbitmq-secret.yaml

 ---
kind: Secret
apiVersion: v1
metadata:
  name: rabbitmq-secret
  namespace: zdevops
data:
  pass: UEA4OHcwcmQ=
  user: YWRtaW4=
type: Opaque

rabbitmq-sts.yaml

 ---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: zdevops
  labels:
    app: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rabbitmq
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rabbitmq
          image: 'registry.zdevops.com.cn/library/rabbitmq:3.9.22-management'
          ports:
            - name: tcp-5672
              containerPort: 5672
              protocol: TCP
            - name: http-15672
              containerPort: 15672
              protocol: TCP
          env:
            - name: RABBITMQ_DEFAULT_USER
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: user
            - name: RABBITMQ_DEFAULT_PASS
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: pass
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 100m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
  serviceName: rabbitmq-headless

---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-headless
  namespace: zdevops
  labels:
    app: rabbitmq
spec:
  ports:
    - name: tcp-rabbitmq-5672
      protocol: TCP
      port: 5672
      targetPort: 5672
  selector:
    app: rabbitmq
  clusterIP: None
  type: ClusterIP

rabbitmq-external.yaml

 ---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-external
  namespace: zdevops
  labels:
    app: rabbitmq-external
spec:
  ports:
    - name: http-rabbitmq-external
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app: rabbitmq
  type: NodePort

GitOps

Operate on the operation and maintenance development server

 # 在已有代码仓库创建 rabbitmq/single 目录
[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/single

# 编辑资源配置清单
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-secret.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-sts.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/single/rabbitmq-external.yaml

# 提交 Git
[root@zdevops-master k8s-yaml]# git add rabbitmq
[root@zdevops-master k8s-yaml]# git commit -am '添加rabbitmq 单节点资源配置清单'
[root@zdevops-master k8s-yaml]# git push

Deploy resources

Operate on the operation and maintenance management server

Update mirror repository code

 [root@zdevops-master k8s-yaml]# git pull

Deploy resources

 [root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/single/

verify

Secret

 [root@zdevops-master k8s-yaml]# kubectl get secret -n zdevops
NAME              TYPE     DATA   AGE
rabbitmq-secret   Opaque   2      8s

StatefulSet

 [root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevops
NAME       READY   AGE   CONTAINERS   IMAGES
rabbitmq   1/1     25s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

Pods

 [root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevops
NAME         READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
rabbitmq-0   1/1     Running   0          26s   10.233.87.13   ks-k8s-master-1   <none>           <none>

Service

 [root@zdevops-master k8s-yaml]# kubectl get svc -o wide -n zdevops
NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE   SELECTOR
rabbitmq-external   NodePort    10.233.4.224   <none>        15672:31672/TCP   36s   app=rabbitmq
rabbitmq-headless   ClusterIP   None           <none>        5672/TCP          36s   app=rabbitmq

Graphical management interface

rabbitmq-management

clean up resources

Clean up StatefulSet

 [root@zdevops-master k8s-yaml]# kubectl delete sts rabbitmq -n zdevops

cleaning service

 [root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-external rabbitmq-headless -n zdevops

Clustered RabbitMQ deployment

Sorting out ideas

Use the official RabbitMQ Cluster Operator for Kubernetes.

Open source RabbitMQ Cluster Kubernetes Operator by VMware.

Quick deployment of official documents

The official provides an example of rapid deployment, which only takes two steps. Here we only use it as a reference. Later, we refer to the offline mirror deployment scheme of the intranet.

Deploy the RabbitMQ Cluster Operator

 kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

Deploy RabbitMQ Cluster

 kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

Prepare for offline mirroring

Perform the following operations on a server that can access both the Internet and the intranet Harbor warehouse.

Create a project in Harbor

 curl -u "admin:Harbor12345" -X POST -H "Content-Type: application/json" https://registry.zdevops.com.cn/api/v2.0/projects -d '{ "project_name": "rabbitmqoperator", "public": true}'

Download mirror

 docker pull rabbitmqoperator/cluster-operator:1.14.0

re-tag

 docker tag rabbitmqoperator/cluster-operator:1.14.0 registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

Push to private mirror repository

 docker push registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

clean up temporary images

 docker rmi rabbitmqoperator/cluster-operator:1.14.0
docker rmi registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0

Resource configuration checklist

Obtain the RabbitMQ Cluster Operator deployment resource configuration list "cluster-operator.yml" from the official website

 wget https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

Modify the RabbitMQ Cluster Operator image to an intranet image

 sed -i 's#rabbitmqoperator#registry.zdevops.com.cn/rabbitmqoperator#g' cluster-operator.yml

RabbitMQ Cluster deployment resource list "rabbitmq-cluster.yaml"

 apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  namespace: zdevops
  name: rabbitmq-cluster
  labels:
    app: rabbitmq-cluster
spec:
  replicas: 3
  image: registry.zdevops.com.cn/library/rabbitmq:3.9.22-management
  resources:
    limits:
      cpu: 2
      memory: 4Gi
    requests:
      cpu: 100m
      memory: 500Mi
  rabbitmq:
    additionalConfig: |
      default_user=admin
      default_pass=P@88w0rd

For more configuration parameters and configuration examples, please refer to the official documentation

The external access service rabbitmq-cluster-external.yaml for the management page

 ---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-cluster-external
  namespace: zdevops
  labels:
    app: rabbitmq-cluster-external
spec:
  ports:
    - name: management
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app.kubernetes.io/name: rabbitmq-cluster
  type: NodePort

GitOps

Operate on the operation and maintenance development server

 # 在已有代码仓库创建 rabbitmq/cluster 目录
[root@zdevops-master k8s-yaml]# mkdir -p rabbitmq/cluster

# 编辑资源配置清单
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/cluster-operator.yml
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster.yaml
[root@zdevops-master k8s-yaml]# vi rabbitmq/cluster/rabbitmq-cluster-external.yaml

# 提交 Git
[root@zdevops-master k8s-yaml]# git add rabbitmq/cluster
[root@zdevops-master k8s-yaml]# git commit -am '添加 rabbitmq 集群模式部署资源配置清单'
[root@zdevops-master k8s-yaml]# git push

Deploy resources

Operate on the operation and maintenance management server

`Update the mirror repository code

 [root@zdevops-master k8s-yaml]# git pull

Deploy the RabbitMQ Cluster Operator

 [root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/cluster-operator.yml

Deploy RabbitMQ Cluster

 [root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster.yaml

Deployment Management Page External Access Services

 [root@zdevops-master k8s-yaml]# kubectl apply -f rabbitmq/cluster/rabbitmq-cluster-external.yaml

verify

RabbitMQ Cluster Operator Deployment

 [root@zdevops-master k8s-yaml]# kubectl get deployments -n rabbitmq-system -o wide
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                             SELECTOR
rabbitmq-cluster-operator   1/1     1            1           107m   operator     registry.zdevops.com.cn/rabbitmqoperator/cluster-operator:1.14.0   app.kubernetes.io/name=rabbitmq-cluster-operator

RabbitmqClusters

 [root@zdevops-master k8s-yaml]# kubectl get rabbitmqclusters -n zdevops
NAME               ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmq-cluster   False              Unknown            23s

StatefulSet

 [root@zdevops-master k8s-yaml]# kubectl get sts -o wide -n zdevops
NAME                      READY   AGE   CONTAINERS   IMAGES
rabbitmq-cluster-server   3/3     74s   rabbitmq     registry.zdevops.com.cn/library/rabbitmq:3.9.22-management

Pods

 [root@zdevops-master k8s-yaml]# kubectl get pods -o wide -n zdevops
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE              NOMINATED NODE   READINESS GATES
rabbitmq-cluster-server-0   1/1     Running   0          84s   10.233.116.26   ks-k8s-master-2   <none>           <none>
rabbitmq-cluster-server-1   1/1     Running   0          83s   10.233.117.28   ks-k8s-master-0   <none>           <none>
rabbitmq-cluster-server-2   1/1     Running   0          82s   10.233.87.31    ks-k8s-master-1   <none>           <none>

Services

 [root@zdevops-master k8s-yaml]# kubectl get svc -n zdevops -o wide
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE    SELECTOR
rabbitmq-cluster            ClusterIP   10.233.56.153   <none>        15692/TCP,5672/TCP,15672/TCP   107s   app.kubernetes.io/name=rabbitmq-cluster
rabbitmq-cluster-external   NodePort    10.233.63.84    <none>        15672:31672/TCP                74m    app.kubernetes.io/name=rabbitmq-cluster
rabbitmq-cluster-nodes      ClusterIP   None            <none>        4369/TCP,25672/TCP             107s   app.kubernetes.io/name=rabbitmq-cluster

Graphical management interface

rabbitmq-management-cluster

From the management interface, you can see a three-node cluster

clean up resources

Clean up RabbitmqClusters

 [root@zdevops-master k8s-yaml]# kubectl delete rabbitmqclusters rabbitmq-cluster -n zdevops

Clean up admin page external services

 [root@zdevops-master k8s-yaml]# kubectl delete svc rabbitmq-cluster-external -n zdevops

concluding remarks

This series of documents are my notes on the learning and operation and maintenance practice in the field of cloud native technology. Using output to force input is an efficient learning method, which can quickly accumulate experience and improve technology. Let others understand, to show that the knowledge is truly mastered.

This series of documents covers (but is not limited to) the following technical areas:

KubeSphere
Kubernetes
Ansible
Automated operation and maintenance
CNCF Technology Stack

If you like this article, please share it with your friends!

Get Document

Github: https://github.com/devops/z-notes
Gitee: https://gitee.com/zdevops/z-notes
Know: https://www.zhihu.com/people/zdevops/

Get code

Github: https://github.com/devops/ansible-zdevops
Gitee: https://gitee.com/zdevops/ansible-zdevops

Get Video Station B

Old Z's Notes https://space.bilibili.com/1039301316

This article is published by OpenWrite , a multi-post blog platform!

Deploy RabbitMQ on K8s cluster in practice

foreword

Introduction

Single node RabbitMQ deployment

Sorting out ideas

Prepare for offline mirroring

Resource configuration checklist

GitOps

Deploy resources

verify

clean up resources

Clustered RabbitMQ deployment

Sorting out ideas

Quick deployment of official documents

Prepare for offline mirroring

Resource configuration checklist

GitOps

Deploy resources

verify

clean up resources

concluding remarks

KubeSphere

引用和评论

云原生周刊：K8s引领潮流

深度求索发布DeepSeek：高效、低成本的开源大语言模型

华为云开天 aPaaS 平台的流使用体验

得物新一代可观测性架构：海量数据下的存算分离设计与实践

宇树 Unitree 推出的 B2-W 机器狗：智能时代的多面 “能手”

英伟达打造个人 AI 超级计算机：Project DIGITS 震撼登场

微短剧出海大升级：窄带高清降本，极致播放体验，海外合规更安心