Helm3 install RabbitMQ cluster

foreword

Recently, when using k8s to build microservices, I found that it is very troublesome to manually modify the pod name, pod image, svc name, ingress tls, etc. in the yaml file, but the situation is different with helm, which is a package of k8s The manager is similar to apt-get of ubuntu and yum of centos. It is very convenient to have the helm installation package. The following explains how to install rabbitmq through helm.

Ready to work

• install k8s
  I am using Alibaba Cloud's ACK k8s service.
• Install k8s client: kubectl
  kubectl installation address
• Install the helm client
  install helm

• Configure the helm repo source
  Here are the three sources I added: stable, bitnami and ali

  helm repo add stable https://charts.helm.sh/stable
  helm repo add bitnami https://charts.helm.sh/stable
  helm repo add ali https://charts.helm.sh/stable

  View the installed repo source

  $ helm repo list                                                    
  NAME       URL
  stable     https://charts.helm.sh/stable
  bitnami    https://charts.bitnami.com/bitnami
  ali        https://apphub.aliyuncs.com/stable/

Rabbitmq installation method

There are many ways to install rabbitmq. Here are a few general installation methods:

• Centos 7/8 install rabbitmq Cloud ECS CentOS provides installation
• k8s installation rabbitmq official document provides installation
• helm installs rabbitmq community boss provides installation

Rabbitmq installation method in different environments

Here we use helm to install rabbitmq, but there is one point we should pay attention to is that if rabbitmq is used as development and testing or pre-production and production environment (under normal circumstances in the company, the development environment and test environment or pre-production environment and The production environment is classified into one category), is the rabbitmq installation method the same? Of course it is different, so let's explain the difference between installing rabbitmq in different environments.

• Install development and test environments (dev, test)

  • k8s Service：type: NodePort | LoadBalance
  • RMQ management interface: use ip:port to access, such as: 192.168.0.1:15672
  • AMQP 5672 port: We also use ip:port to access, such as: 192.168.0.1:5672

• Install pre-production and production environments (uat, prod)

  • k8s Service：type: ClusterIP，Ingress
  • RMQ management interface: use the domain name configured in ingress to access, such as: rabbitmq.demo.com
  • AMQP port 5672: use the name resolved by k8s internet dns to access, such as: test-rabbitmq-headless.rabbit.svc.cluster.local:5672

Let's explain how to install rabbitmq in different environments.

Preparation before installation

First, let's see if there is a rabbitmq chart in helm,

$ helm search repo rabbitmq
NAME                                   CHART VERSION    APP VERSION    DESCRIPTION
ali/prometheus-rabbitmq-exporter       0.5.5            v0.29.0        Rabbitmq metrics exporter for prometheus
ali/rabbitmq                           6.18.2           3.8.2          DEPRECATED Open source message broker software ...
ali/rabbitmq-ha                        1.47.0           3.8.7          Highly available RabbitMQ cluster, the open sou...
bitnami/rabbitmq                       8.16.1           3.8.18         Open source message broker software that implem...
stable/prometheus-rabbitmq-exporter    0.5.6            v0.29.0        DEPRECATED Rabbitmq metrics exporter for promet...
stable/rabbitmq                        6.18.2           3.8.2          DEPRECATED Open source message broker software ...
stable/rabbitmq-ha                     1.47.1           3.8.7          DEPRECATED - Highly available RabbitMQ cluster,...

It can be seen that the version of the rabbitmq chart provided by different repo sources is also different. We choose bitnami/rabbitmq, chart version: 8.16.1, APP version: 3.8.18.

Next, we need to download the stable/rabbitmq chart file. The downloaded chart file is a .tgz compressed file. Just decompress it.

helm pull bitnami/rabbitmq
tar zxf rabbitmq-8.16.1.tgz

$ cd rabbitmq && ls -ls
total 168
 8 -rwxr-xr-x   1 zhangwei  staff    435  1  1  1970 Chart.yaml
72 -rwxr-xr-x   1 zhangwei  staff  34706  1  1  1970 README.md
 0 drwxr-xr-x   5 zhangwei  staff    160  6 30 14:43 ci
 0 drwxr-xr-x  19 zhangwei  staff    608  6 30 14:43 templates
40 -rwxr-xr-x   1 zhangwei  staff  19401  1  1  1970 values-production.yaml
 8 -rwxr-xr-x   1 zhangwei  staff   2854  1  1  1970 values.schema.json
40 -rwxr-xr-x   1 zhangwei  staff  18986  1  1  1970 values.yaml

Then, let's look at the available parameters provided by values.yaml, or through the command:

helm show values bitnami/rabbitmq

values.yaml

## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
#   imageRegistry: myRegistryName
#   imagePullSecrets:
#     - myRegistryKeySecretName
#   storageClass: myStorageClass

## Bitnami RabbitMQ image version
## ref: https://hub.docker.com/r/bitnami/rabbitmq/tags/
##
image:
  registry: docker.io
  repository: bitnami/rabbitmq
  tag: 3.8.2-debian-10-r30

  ## set to true if you would like to see extra information on logs
  ## it turns BASH and NAMI debugging in minideb
  ## ref:  https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
  debug: false

  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistryKeySecretName

## String to partially override rabbitmq.fullname template (will maintain the release name)
##
# nameOverride:

## String to fully override rabbitmq.fullname template
##
# fullnameOverride:

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## does your cluster have rbac enabled? assume yes by default
rbacEnabled: true

## RabbitMQ should be initialized one by one when building cluster for the first time.
## Therefore, the default value of podManagementPolicy is 'OrderedReady'
## Once the RabbitMQ participates in the cluster, it waits for a response from another
## RabbitMQ in the same cluster at reboot, except the last RabbitMQ of the same cluster.
## If the cluster exits gracefully, you do not need to change the podManagementPolicy
## because the first RabbitMQ of the statefulset always will be last of the cluster.
## However if the last RabbitMQ of the cluster is not the first RabbitMQ due to a failure,
## you must change podManagementPolicy to 'Parallel'.
## ref : https://www.rabbitmq.com/clustering.html#restarting
##
podManagementPolicy: OrderedReady

## section of specific values for rabbitmq
rabbitmq:
  ## RabbitMQ application username
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  username: user

  ## RabbitMQ application password
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # password:
  # existingPasswordSecret: name-of-existing-secret

  ## Erlang cookie to determine whether different nodes are allowed to communicate with each other
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # erlangCookie:
  # existingErlangSecret: name-of-existing-secret

  ## Node name to cluster with. e.g.: `clusternode@hostname`
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # rabbitmqClusterNodeName:

  ## Value for the RABBITMQ_LOGS environment variable
  ## ref: https://www.rabbitmq.com/logging.html#log-file-location
  ##
  logs: '-'

  ## RabbitMQ Max File Descriptors
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ## ref: https://www.rabbitmq.com/install-debian.html#kernel-resource-limits
  ##
  setUlimitNofiles: true
  ulimitNofiles: '65536'

  ## RabbitMQ maximum available scheduler threads and online scheduler threads
  ## ref: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html#scheduler-threads
  ##
  maxAvailableSchedulers: 2
  onlineSchedulers: 1

  ## Plugins to enable
  plugins: "rabbitmq_management rabbitmq_peer_discovery_k8s"

  ## Extra plugins to enable
  ## Use this instead of `plugins` to add new plugins
  extraPlugins: "rabbitmq_auth_backend_ldap"

  ## Clustering settings
  clustering:
    address_type: hostname
    k8s_domain: cluster.local
    ## Rebalance master for queues in cluster when new replica is created
    ## ref: https://www.rabbitmq.com/rabbitmq-queues.8.html#rebalance
    rebalance: false

  loadDefinition:
    enabled: false
    secretName: load-definition

  ## environment variables to configure rabbitmq
  ## ref: https://www.rabbitmq.com/configure.html#customise-environment
  env: {}

  ## Configuration file content: required cluster configuration
  ## Do not override unless you know what you are doing. To add more configuration, use `extraConfiguration` of `advancedConfiguration` instead
  configuration: |-
    ## Clustering
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    # queue master locator
    queue_master_locator=min-masters
    # enable guest user
    loopback_users.guest = false

  ## Configuration file content: extra configuration
  ## Use this instead of `configuration` to add more configuration
  extraConfiguration: |-
    #disk_free_limit.absolute = 50MB
    #management.load_definitions = /app/load_definition.json

  ## Configuration file content: advanced configuration
  ## Use this as additional configuraton in classic config format (Erlang term configuration format)
  ##
  ## If you set LDAP with TLS/SSL enabled and you are using self-signed certificates, uncomment these lines.
  ## advancedConfiguration: |-
  ##   [{
  ##     rabbitmq_auth_backend_ldap,
  ##     [{
  ##         ssl_options,
  ##         [{
  ##             verify, verify_none
  ##         }, {
  ##             fail_if_no_peer_cert,
  ##             false
  ##         }]
  ##     ]}
  ##   }].
  ##
  advancedConfiguration: |-

  ## Enable encryption to rabbitmq
  ## ref: https://www.rabbitmq.com/ssl.html
  ##
  tls:
    enabled: false
    failIfNoPeerCert: true
    sslOptionsVerify: verify_peer
    caCertificate: |-
    serverCertificate: |-
    serverKey: |-
    # existingSecret: name-of-existing-secret-to-rabbitmq

## LDAP configuration
##
ldap:
  enabled: false
  server: ""
  port: "389"
  user_dn_pattern: cn=${username},dc=example,dc=org
  tls:
    # If you enabled TLS/SSL you can set advaced options using the advancedConfiguration parameter.
    enabled: false

## Kubernetes service type
service:
  type: ClusterIP
  ## Node port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # nodePort: 30672

  ## Set the LoadBalancerIP
  ##
  # loadBalancerIP:

  ## Node port Tls
  ##
  # nodeTlsPort: 30671

  ## Amqp port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  port: 5672

  ## Amqp Tls port
  ##
  tlsPort: 5671

  ## Dist port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  distPort: 25672

  ## RabbitMQ Manager port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  managerPort: 15672

  ## Service annotations
  annotations: {}
    # service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

  ## Load Balancer sources
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
  ##
  # loadBalancerSourceRanges:
  # - 10.10.10.0/24

  ## Extra ports to expose
  # extraPorts:

  ## Extra ports to be included in container spec, primarily informational
  # extraContainerPorts:

# Additional pod labels to apply
podLabels: {}

## Pod Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001
  extra: {}

persistence:
  ## this enables PVC templates that will create one per pod
  enabled: true

  ## rabbitmq data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  accessMode: ReadWriteOnce

  ## Existing PersistentVolumeClaims
  ## The value is evaluated as a template
  ## So, for example, the name can depend on .Release or .Chart
  # existingClaim: ""

  # If you change this value, you might have to adjust `rabbitmq.diskFreeLimit` as well.
  size: 8Gi

  # persistence directory, maps to the rabbitmq data directory
  path: /opt/bitnami/rabbitmq/var/lib/rabbitmq

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}

networkPolicy:
  ## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/
  ##
  enabled: false

  ## The Policy model to apply. When set to false, only pods with the correct
  ## client label will have network access to the ports RabbitMQ is listening
  ## on. When true, RabbitMQ will accept connections from any source
  ## (with the correct destination port).
  ##
  allowExternal: true

  ## Additional NetworkPolicy Ingress "from" rules to set. Note that all rules are OR-ed.
  ##
  # additionalRules:
  #  - matchLabels:
  #    - role: frontend
  #  - matchExpressions:
  #    - key: role
  #      operator: In
  #      values:
  #        - frontend

## Replica count, set to 1 to provide a default available cluster
replicas: 1

## Pod priority
## https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
# priorityClassName: ""

## updateStrategy for RabbitMQ statefulset
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategy:
  type: RollingUpdate

## Node labels and tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
nodeSelector: {}
tolerations: []
affinity: {}
podDisruptionBudget: {}
  # maxUnavailable: 1
  # minAvailable: 1
## annotations for rabbitmq pods
podAnnotations: {}

## Configure the ingress resource that allows you to access the
## Wordpress installation. Set up the URL
## ref: http://kubernetes.io/docs/user-guide/ingress/
##
ingress:
  ## Set to true to enable ingress record generation
  enabled: false

  ## The list of hostnames to be covered with this ingress record.
  ## Most likely this will be just one host, but in the event more hosts are needed, this is an array
  ## hostName: foo.bar.com
  path: /

  ## Set this to true in order to enable TLS on the ingress record
  ## A side effect of this will be that the backend wordpress service will be connected at port 443
  tls: false

  ## If TLS is set to true, you must declare what secret will store the key/certificate for TLS
  tlsSecret: myTlsSecret

  ## Ingress annotations done as key:value pairs
  ## If you're using kube-lego, you will want to add:
  ## kubernetes.io/tls-acme: true
  ##
  ## For a full list of possible ingress annotations, please see
  ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
  ##
  ## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set
  annotations: {}
  #  kubernetes.io/ingress.class: nginx
  #  kubernetes.io/tls-acme: true

## The following settings are to configure the frequency of the lifeness and readiness probes
livenessProbe:
  enabled: true
  initialDelaySeconds: 120
  timeoutSeconds: 20
  periodSeconds: 30
  failureThreshold: 6
  successThreshold: 1

readinessProbe:
  enabled: true
  initialDelaySeconds: 10
  timeoutSeconds: 20
  periodSeconds: 30
  failureThreshold: 3
  successThreshold: 1

metrics:
  enabled: false
  image:
    registry: docker.io
    repository: bitnami/rabbitmq-exporter
    tag: 0.29.0-debian-10-r28
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  ## environment variables to configure rabbitmq_exporter
  ## ref: https://github.com/kbudde/rabbitmq_exporter#configuration
  env: {}
  ## Metrics exporter port
  port: 9419
  ## RabbitMQ address to connect to (from the same Pod, usually the local loopback address).
  ## If your Kubernetes cluster does not support IPv6, you can change to `127.0.0.1` in order to force IPv4.
  ## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#networking
  rabbitmqAddress: localhost
  ## Comma-separated list of extended scraping capabilities supported by the target RabbitMQ server
  ## ref: https://github.com/kbudde/rabbitmq_exporter#extended-rabbitmq-capabilities
  capabilities: "bert,no_sort"
  resources: {}
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9419"

  livenessProbe:
    enabled: true
    initialDelaySeconds: 15
    timeoutSeconds: 5
    periodSeconds: 30
    failureThreshold: 6
    successThreshold: 1

  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    timeoutSeconds: 5
    periodSeconds: 30
    failureThreshold: 3
    successThreshold: 1

  ## Prometheus Service Monitor
  ## ref: https://github.com/coreos/prometheus-operator
  ##      https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
  serviceMonitor:
    ## If the operator is installed in your cluster, set to true to create a Service Monitor Entry
    enabled: false
    ## Specify the namespace in which the serviceMonitor resource will be created
    # namespace: ""
    ## Specify the interval at which metrics should be scraped
    interval: 30s
    ## Specify the timeout after which the scrape is ended
    # scrapeTimeout: 30s
    ## Specify Metric Relabellings to add to the scrape endpoint
    # relabellings:
    ## Specify honorLabels parameter to add the scrape endpoint
    honorLabels: false
    ## Specify the release for ServiceMonitor. Sometimes it should be custom for prometheus operator to work
    # release: ""
    ## Used to pass Labels that are used by the Prometheus installed in your cluster to select Service Monitors to work with
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
    additionalLabels: {}

  ## Custom PrometheusRule to be defined
  ## The value is evaluated as a template, so, for example, the value can depend on .Release or .Chart
  ## ref: https://github.com/coreos/prometheus-operator#customresourcedefinitions
  prometheusRule:
    enabled: false
    additionalLabels: {}
    namespace: ""
    rules: []
      ## List of reules, used as template by Helm.
      ## These are just examples rules inspired from https://awesome-prometheus-alerts.grep.to/rules.html
      ## Please adapt them to your needs.
      ## Make sure to constraint the rules to the current rabbitmq service.
      ## Also make sure to escape what looks like helm template.
      # - alert: RabbitmqDown
      #   expr: rabbitmq_up{service="{{ template "rabbitmq.fullname" . }}"} == 0
      #   for: 5m
      #   labels:
      #     severity: error
      #   annotations:
      #     summary: Rabbitmq down (instance {{ "{{ $labels.instance }}" }})
      #     description: RabbitMQ node down

      # - alert: ClusterDown
      #   expr: |
      #     sum(rabbitmq_running{service="{{ template "rabbitmq.fullname" . }}"})
      #     < {{ .Values.replicas }}
      #   for: 5m
      #   labels:
      #     severity: error
      #   annotations:
      #     summary: Cluster down (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         Less than {{ .Values.replicas }} nodes running in RabbitMQ cluster
      #         VALUE = {{ "{{ $value }}" }}

      # - alert: ClusterPartition
      #   expr: rabbitmq_partitions{service="{{ template "rabbitmq.fullname" . }}"} > 0
      #   for: 5m
      #   labels:
      #     severity: error
      #   annotations:
      #     summary: Cluster partition (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         Cluster partition
      #         VALUE = {{ "{{ $value }}" }}

      # - alert: OutOfMemory
      #   expr: |
      #     rabbitmq_node_mem_used{service="{{ template "rabbitmq.fullname" . }}"}
      #     / rabbitmq_node_mem_limit{service="{{ template "rabbitmq.fullname" . }}"}
      #     * 100 > 90
      #   for: 5m
      #   labels:
      #     severity: warning
      #   annotations:
      #     summary: Out of memory (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         Memory available for RabbmitMQ is low (< 10%)\n  VALUE = {{ "{{ $value }}" }}
      #         LABELS: {{ "{{ $labels }}" }}

      # - alert: TooManyConnections
      #   expr: rabbitmq_connectionsTotal{service="{{ template "rabbitmq.fullname" . }}"} > 1000
      #   for: 5m
      #   labels:
      #     severity: warning
      #   annotations:
      #     summary: Too many connections (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         RabbitMQ instance has too many connections (> 1000)
      #         VALUE = {{ "{{ $value }}" }}\n  LABELS: {{ "{{ $labels }}" }}

##
## Init containers parameters:
## volumePermissions: Change the owner of the persist volume mountpoint to RunAsUser:fsGroup
##
volumePermissions:
  enabled: false
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: buster
    pullPolicy: Always
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  resources: {}

## forceBoot: executes 'rabbitmqctl force_boot' to force boot cluster shut down unexpectedly in an
## unknown order.
## ref: https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot
##
forceBoot:
  enabled: false

## Optionally specify extra secrets to be created by the chart.
## This can be useful when combined with load_definitions to automatically create the secret containing the definitions to be loaded.
##
extraSecrets: {}
  # load-definition:
  #   load_definition.json: |
  #     {
  #       ...
  #     }

The values.yaml file is the configuration information disclosed by the helm chart, which can be modified as needed. Because we are using Alibaba Cloud, we use alicloud-disk-ssd for storage. Note that the minimum requirement for alicloud-disk-ssd is 20G:

storageClass: "alicloud-disk-ssd"
size: 20Gi

Next, we create the rabbit namespace

kubectl create namespace rabbit

Installation in development and test environment

Because we need to use ip:port to access the development environment, we need to configure the service.
The final complete values.yaml file is as follows:

values.yaml

service:
  type: NodePort
persistence:
  storageClass: "alicloud-disk-ssd"
  size: 20Gi

Now let's install rabbitmq by running the following command:

# 创建rabbitmq集群
helm install -f values.yaml test-rabbitmq bitnami/rabbitmq --namespace rabbit

Here is the output when installing:

output starts

---

$ helm install -f test-values.yaml test-rabbitmq bitnami/rabbitmq --namespace rabbit
NAME: test-rabbitmq
LAST DEPLOYED: Thu Jul  8 10:07:50 2021
NAMESPACE: rabbit
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Credentials:
    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"
    echo "ErLang Cookie : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)"

Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases.

RabbitMQ can be accessed within the cluster on port  at test-rabbitmq.rabbit.svc.

To access for outside the cluster, perform the following steps:

Obtain the NodePort IP and ports:

    export NODE_IP=$(kubectl get nodes --namespace rabbit -o jsonpath="{.items[0].status.addresses[0].address}")
    export NODE_PORT_AMQP=$(kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq)
    export NODE_PORT_STATS=$(kubectl get --namespace rabbit -o jsonpath="{.spec.ports[3].nodePort}" services test-rabbitmq)

To Access the RabbitMQ AMQP port:

    echo "URL : amqp://$NODE_IP:$NODE_PORT_AMQP/"

To Access the RabbitMQ Management interface:

    echo "URL : http://$NODE_IP:$NODE_PORT_STATS/"

end of output

---

Check all rabbitmq resources under the rabbit namespace, whether the creation is successful!

$ kubectl get all -n rabbit
NAME                  READY   STATUS    RESTARTS   AGE
pod/test-rabbitmq-0   1/1     Running   0          3h56m

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                 AGE
service/test-rabbitmq            ClusterIP   172.21.15.81   <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m
service/test-rabbitmq-headless   ClusterIP   None           <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m

NAME                             READY   AGE
statefulset.apps/test-rabbitmq   1/1     3h56m

After waiting for a while, you will find that pod, svc, pvc, pv, and statefulset are all created. How do we access the Rabbitmq management interface? From the output we find:

• NODE_IP
  Command: kubectl get nodes --namespace rabbit -o jsonpath="{.items[0].status.addresses[0].address}"
• NODE_PORT_AMQP
  Command: kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq
• NODE_PORT_STATS
  Command: kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq

The Node_ip above is actually the real ip of the k8s node (see if it is accessible on the public network, if not, you can apply for Alibaba Cloud EIP to bind it to the public network), such as: 192.168.0.1, and NODE_PORT_AMQP is NodePort port corresponding to cluster 5672, such as 31020, NODE_PORT_STATS is the NodePort port , such as 31010
In this way, we access the Rabbitmq management interface and enter: http://192.168.0.1:31010 in the browser to access:

In this way, springboot can access Rabbitmq through the ip address,

spring:
  rabbitmq:
    host: 192.168.0.1
    port: 5672
    username: user
    password:

Installation in pre-production production environment

Because we need to access through the domain name for development, we need to enable ingress and domain name configuration:

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hostName: rabbitmq.demo.com

Finally, to access through https, we need to enable TLS:

tls: true
tlsSecret: tls-secret-name

Let's pay attention here, because I am using the cert-manager.io/cluster-issuer annotation (I have already generated it in k8s), so I can directly generate the TLS certificate, which is very convenient. If you are interested, you can take a look at . cert-manager to apply for a free HTTPS certificate

annotations:
    cert-manager.io/cluster-issuer: your-cert-manager-name

The final complete values.yaml file is as follows:

values.yaml

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: your-cert-manager-name
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
  hostName: rabbitmq.demo.com
  tls: true
  tlsSecret: tls-secret-name
persistence:
  storageClass: "alicloud-disk-ssd"
  size: 20Gi

Now let's install rabbitmq by running the following command:

# 创建rabbitmq集群
helm install -f values.yaml test-rabbitmq bitnami/rabbitmq --namespace rabbit

Here is the output when installing:

output starts

---

NAME: test-rabbitmq
LAST DEPLOYED: Thu Jul  8 19:06:58 2021
NAMESPACE: rabbit
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Credentials:
    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"
    echo "ErLang Cookie : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)"

Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases.

RabbitMQ can be accessed within the cluster on port  at test-rabbitmq.rabbit.svc.

To access for outside the cluster, perform the following steps:

To Access the RabbitMQ AMQP port:

1. Create a port-forward to the AMQP port:

    kubectl port-forward --namespace rabbit svc/test-rabbitmq 5672:5672 &
    echo "URL : amqp://127.0.0.1:5672/"

2. Access RabbitMQ using using the obtained URL.

To Access the RabbitMQ Management interface:

1. Get the RabbitMQ Management URL and associate its hostname to your cluster external IP:

   export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
   echo "RabbitMQ Management: http://rabbitmq.demo.com/"
   echo "$CLUSTER_IP  rabbitmq.testapi.seaurl.com" | sudo tee -a /etc/hosts

2. Open a browser and access RabbitMQ Management using the obtained URL.

end of output

---

Check all rabbitmq resources under the rabbit namespace, whether the creation is successful!

$ kubectl get all -n rabbit
NAME                  READY   STATUS    RESTARTS   AGE
pod/test-rabbitmq-0   1/1     Running   0          3h56m

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                 AGE
service/test-rabbitmq            ClusterIP   172.21.15.81   <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m
service/test-rabbitmq-headless   ClusterIP   None           <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m

NAME                             READY   AGE
statefulset.apps/test-rabbitmq   1/1     3h56m

After waiting for a while, you will find that pod, svc, ingress, pvc, pv, and statefulset are all created. How do we access the Rabbitmq management interface? From the output we find http://rabbitmq.demo.com/
Address, in this way, we access the Rabbitmq management interface and enter: http://rabbitmq.demo.com/ in the browser to complete the access:

In this way, springboot can access Rabbitmq through the name parsed by k8s intranet dns. How to parse it can to this article

spring:
  rabbitmq:
    host: test-rabbitmq-headless.rabbit.svc.cluster.local
    port: 5672
    username: user
    password:

question

1. Access rabbitmq report: 503
If the domain name path you configured is such as: demo.com/rabbitmq , such a domain name, then you must configure it as follows to access it correctly. In addition: recommended to use the first-level or second-level or third-level domain name as the hostName. Do not use path:/rabbitmq. For the form refer to this article :

rabbitmq:
  extraConfiguration: |-
    management.path_prefix = /rabbitmq/
ingress:
...
  hostName: demo.com
  path: /rabbitmq/
...

2. kubectl describe svc your-service-name -n rabbit found that the service endpoint is empty

• Reason: pvc did not delete refer to this article to understand

• solve:

  kubectl get pvc
  kubectl delete pvc <name>

3、running "VolumeBinding" filter plugin for pod "test-rabbitmq-0": pod has unbound immediate PersistentVolumeClaims

• Reason: To set storageClass and Alibaba cloud disk, the minimum requirement is 20Gi

• solve:

  persistence:
  storageClass: "alicloud-disk-ssd"
  size: 20Gi

4、PersistentVolumeClaim "data-test-rabbitmq-0" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims

• Reason: pvc container does not support online modification
• Solution: delete pvc and recreate

5、Warning ProvisioningFailed 6s (x4 over 14m) diskplugin.csi.alibabacloud.com_iZbp1d2cbgi4jt9oty4m9iZ_3408e051-98e8-4295-8a21-7f1af0807958 (combined from similar events): failed to provision volume with StorageClass "alicloud-disk-ssd": rpc error: code = Internal desc = SDK.ServerError
ErrorCode: InvalidAccountStatus.NotEnoughBalance

• Reason: The Alibaba Cloud account balance must be at least 100 yuan to create an SSD cloud disk
• Solution: Aliyun account recharge

6. Log in to rabbitmq and report 401
Reason: wrong password
Solution: Run the following command to get the password

echo "Password      : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"

7. The springcloud microservice accesses port 5672 connection report: time out
Reason: The spring.rabbitmq.host address should not be your ingress external domain name: rabbitmq.demo.com, but should be the address resolved by your intranet cluster dns
Solution: Use the following command to obtain the intranet resolution address of test-rabbitmq, and then assign it to spring.rabbitmq.host to reconnect

# 如果 pod 和服务的命名空间不相同，则 DNS 查询必须指定服务所在的命名空间。
# test-rabbitmq-headless.rabbit命名空间是rabbit
# 如果没有指定命名空间，可以使用kubectl exec -i -t dnsutils -- nslookup test-rabbitmq-headless即可
$ kubectl exec -i -t dnsutils -- nslookup test-rabbitmq-headless.rabbit
Server:        172.21.0.10
Address:    172.21.0.10#53

Name:    test-rabbitmq-headless.rabbit.svc.cluster.local
Address: 172.20.0.124

application.yml

spring.rabbitmq.host=test-rabbitmq-headless.rabbit.svc.cluster.local

8. The NodePort corresponding to 5672 obtained in the development and test environment is incorrect
Reason: The 5672 port obtained by the following command is incorrect

kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq

Solution: Use the following command to view the NodePort port 31972 corresponding to 5672

$ kubectl get svc test-rabbitmq -n rabbit
NAME            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                         AGE
test-rabbitmq   NodePort   172.21.14.17   <none>        5672:31972/TCP,4369:31459/TCP,25672:31475/TCP,15672:31696/TCP   14m

Summarize

1. kubectl needs to configure the connection configuration information of k8s before it can be used, and helm gets the connection information by default, which is in ~/.kube/config by default
2. helm3 has removed the helm tillers and helm init commands, so don't discuss helm2.
3. If an error occurs during or after the installation, you can delete it through the application->helm release list under the Alibaba Cloud ack console.
4. There is no need to configure rabbitmq-plugins enable rabbitmq_management , because helm starts by default after installing rabbitmq.
5. If you want to debug locally, you can use the following methods

To Access the RabbitMQ AMQP port:

    kubectl port-forward --namespace rabbit svc/test-rabbitmq 5672:5672
    echo "URL : amqp://127.0.0.1:5672/"

To Access the RabbitMQ Management interface:

    kubectl port-forward --namespace rabbit svc/test-rabbitmq 15672:15672
    echo "URL : http://127.0.0.1:15672/"

6. The rabbitmq port 5672 only supports the amqp: protocol, which means that it cannot be exposed by ingress, and ingress can only expose http and https, so we set the type of service in the development and test environment: NodePort is the reason! ! !

2022-01-18 Update

Generally, if you set rabbmit.service.type=NodePort , if it is a pod in the cluster, you can directly access it through the dns cluster domain name, such as: host: test-rabbitmq，headless.rabbit.svc.cluster.local，port: 5672 , but if you want to access the graphical interface through the external network, how to do it?
Solution: Because our rabbitmq is a stateful deployment of StatefulSet, the IP on the node is unchanged. Based on this, we can directly obtain the IP of the Node node and specify the external port of 15672 to access it, provided that the Node security group requires Release the external port corresponding to 15672.
For example, 15672 below corresponds to port 31961, so that you can access the graphical interface.

service/test-rabbitmq            NodePort    172.21.10.248   <none>        5672:31469/TCP,4369:32745/TCP,25672:32057/TCP,15672:31961/TCP   183d

quote

Helm deploys RabbitMQ
helm documentation
How does kubectl exec work?
Helm Tutorial