Helm install RabbitMQ cluster

Awbeci
中文

Preface

Recently, when using k8s to build microservices, I found that you need to manually modify the pod name, pod image, svc name, ingress tls, etc. in the yaml file, which is very troublesome, but the situation is different with helm. Helm is a package of k8s Manager, similar to apt-get of ubuntu and yum of centos, it is very convenient to have the helm installation package. The following explains how to install rabbitmq through helm.

Ready to work

  • Install k8s
    I am using Alibaba Cloud's ACK k8s service.
  • Install k8s client: kubectl
    kubectl installation address
  • Install the helm client
    install helm
  • Configure helm repo source
    Here are the three sources I added: stable, bitnami and ali

    helm repo add stable https://charts.helm.sh/stable
    helm repo add bitnami https://charts.helm.sh/stable
    helm repo add ali https://charts.helm.sh/stable

    View the installed repo source

    $ helm repo list                                                    
    NAME       URL
    stable     https://charts.helm.sh/stable
    bitnami    https://charts.bitnami.com/bitnami
    ali        https://apphub.aliyuncs.com/stable/

How to install Rabbitmq

There are many ways to install rabbitmq. Here are a few conventional installation methods:

How to install Rabbitmq in different environments

Here we use helm to install rabbitmq, but one thing we should pay attention to is that if rabbitmq is used as a development and testing or pre-production and production environment (in general, the company will use the development environment and the test environment or the pre-production environment and The production environment is classified as one category), is the rabbitmq installation method the same? Of course it is different, so below we explain the difference between installing rabbitmq in different environments.

  • Install development and test environment (dev, test)

    • k8s Service:type: NodePort | LoadBalance
    • RMQ management interface: use ip:port to access, such as: 192.168.0.1:15672
    • AMQP 5672 port: We also use ip:port to access, such as: 192.168.0.1:5672
  • install pre-production and production environment (uat, prod)

    • k8s Service:type: ClusterIP,Ingress
    • RMQ management interface: use the domain name configured in ingress to access, such as rabbitmq.demo.com
    • AMQP 5672 port: use the name resolved by k8s internet dns to access, such as: test-rabbitmq-headless.rabbit.svc.cluster.local:5672

Below we will explain how rabbitmq is installed in different environments.

Preparation before installation

First, let's see if there is a rabbitmq chart in helm,

$ helm search repo rabbitmq
NAME                                   CHART VERSION    APP VERSION    DESCRIPTION
ali/prometheus-rabbitmq-exporter       0.5.5            v0.29.0        Rabbitmq metrics exporter for prometheus
ali/rabbitmq                           6.18.2           3.8.2          DEPRECATED Open source message broker software ...
ali/rabbitmq-ha                        1.47.0           3.8.7          Highly available RabbitMQ cluster, the open sou...
bitnami/rabbitmq                       8.16.1           3.8.18         Open source message broker software that implem...
stable/prometheus-rabbitmq-exporter    0.5.6            v0.29.0        DEPRECATED Rabbitmq metrics exporter for promet...
stable/rabbitmq                        6.18.2           3.8.2          DEPRECATED Open source message broker software ...
stable/rabbitmq-ha                     1.47.1           3.8.7          DEPRECATED - Highly available RabbitMQ cluster,...

You can see that the rabbitmq chart versions provided by different repo sources are also different. We chose bitnami/rabbitmq, chart version: 8.16.1, APP version: 3.8.18.

Next, we need to download the stable/rabbitmq chart file, the downloaded chart file is a compressed file of .tgz, just unzip it,

helm pull bitnami/rabbitmq
tar zxf rabbitmq-8.16.1.tgz
$ cd rabbitmq && ls -ls
total 168
 8 -rwxr-xr-x   1 zhangwei  staff    435  1  1  1970 Chart.yaml
72 -rwxr-xr-x   1 zhangwei  staff  34706  1  1  1970 README.md
 0 drwxr-xr-x   5 zhangwei  staff    160  6 30 14:43 ci
 0 drwxr-xr-x  19 zhangwei  staff    608  6 30 14:43 templates
40 -rwxr-xr-x   1 zhangwei  staff  19401  1  1  1970 values-production.yaml
 8 -rwxr-xr-x   1 zhangwei  staff   2854  1  1  1970 values.schema.json
40 -rwxr-xr-x   1 zhangwei  staff  18986  1  1  1970 values.yaml

Then, let's check the available parameters externally provided by values.yaml, or through the command:

helm show values bitnami/rabbitmq

values.yaml

## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
#   imageRegistry: myRegistryName
#   imagePullSecrets:
#     - myRegistryKeySecretName
#   storageClass: myStorageClass

## Bitnami RabbitMQ image version
## ref: https://hub.docker.com/r/bitnami/rabbitmq/tags/
##
image:
  registry: docker.io
  repository: bitnami/rabbitmq
  tag: 3.8.2-debian-10-r30

  ## set to true if you would like to see extra information on logs
  ## it turns BASH and NAMI debugging in minideb
  ## ref:  https://github.com/bitnami/minideb-extras/#turn-on-bash-debugging
  debug: false

  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistryKeySecretName

## String to partially override rabbitmq.fullname template (will maintain the release name)
##
# nameOverride:

## String to fully override rabbitmq.fullname template
##
# fullnameOverride:

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## does your cluster have rbac enabled? assume yes by default
rbacEnabled: true

## RabbitMQ should be initialized one by one when building cluster for the first time.
## Therefore, the default value of podManagementPolicy is 'OrderedReady'
## Once the RabbitMQ participates in the cluster, it waits for a response from another
## RabbitMQ in the same cluster at reboot, except the last RabbitMQ of the same cluster.
## If the cluster exits gracefully, you do not need to change the podManagementPolicy
## because the first RabbitMQ of the statefulset always will be last of the cluster.
## However if the last RabbitMQ of the cluster is not the first RabbitMQ due to a failure,
## you must change podManagementPolicy to 'Parallel'.
## ref : https://www.rabbitmq.com/clustering.html#restarting
##
podManagementPolicy: OrderedReady

## section of specific values for rabbitmq
rabbitmq:
  ## RabbitMQ application username
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  username: user

  ## RabbitMQ application password
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # password:
  # existingPasswordSecret: name-of-existing-secret

  ## Erlang cookie to determine whether different nodes are allowed to communicate with each other
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # erlangCookie:
  # existingErlangSecret: name-of-existing-secret

  ## Node name to cluster with. e.g.: `clusternode@hostname`
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # rabbitmqClusterNodeName:

  ## Value for the RABBITMQ_LOGS environment variable
  ## ref: https://www.rabbitmq.com/logging.html#log-file-location
  ##
  logs: '-'

  ## RabbitMQ Max File Descriptors
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ## ref: https://www.rabbitmq.com/install-debian.html#kernel-resource-limits
  ##
  setUlimitNofiles: true
  ulimitNofiles: '65536'

  ## RabbitMQ maximum available scheduler threads and online scheduler threads
  ## ref: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html#scheduler-threads
  ##
  maxAvailableSchedulers: 2
  onlineSchedulers: 1

  ## Plugins to enable
  plugins: "rabbitmq_management rabbitmq_peer_discovery_k8s"

  ## Extra plugins to enable
  ## Use this instead of `plugins` to add new plugins
  extraPlugins: "rabbitmq_auth_backend_ldap"

  ## Clustering settings
  clustering:
    address_type: hostname
    k8s_domain: cluster.local
    ## Rebalance master for queues in cluster when new replica is created
    ## ref: https://www.rabbitmq.com/rabbitmq-queues.8.html#rebalance
    rebalance: false

  loadDefinition:
    enabled: false
    secretName: load-definition

  ## environment variables to configure rabbitmq
  ## ref: https://www.rabbitmq.com/configure.html#customise-environment
  env: {}

  ## Configuration file content: required cluster configuration
  ## Do not override unless you know what you are doing. To add more configuration, use `extraConfiguration` of `advancedConfiguration` instead
  configuration: |-
    ## Clustering
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    # queue master locator
    queue_master_locator=min-masters
    # enable guest user
    loopback_users.guest = false

  ## Configuration file content: extra configuration
  ## Use this instead of `configuration` to add more configuration
  extraConfiguration: |-
    #disk_free_limit.absolute = 50MB
    #management.load_definitions = /app/load_definition.json

  ## Configuration file content: advanced configuration
  ## Use this as additional configuraton in classic config format (Erlang term configuration format)
  ##
  ## If you set LDAP with TLS/SSL enabled and you are using self-signed certificates, uncomment these lines.
  ## advancedConfiguration: |-
  ##   [{
  ##     rabbitmq_auth_backend_ldap,
  ##     [{
  ##         ssl_options,
  ##         [{
  ##             verify, verify_none
  ##         }, {
  ##             fail_if_no_peer_cert,
  ##             false
  ##         }]
  ##     ]}
  ##   }].
  ##
  advancedConfiguration: |-

  ## Enable encryption to rabbitmq
  ## ref: https://www.rabbitmq.com/ssl.html
  ##
  tls:
    enabled: false
    failIfNoPeerCert: true
    sslOptionsVerify: verify_peer
    caCertificate: |-
    serverCertificate: |-
    serverKey: |-
    # existingSecret: name-of-existing-secret-to-rabbitmq

## LDAP configuration
##
ldap:
  enabled: false
  server: ""
  port: "389"
  user_dn_pattern: cn=${username},dc=example,dc=org
  tls:
    # If you enabled TLS/SSL you can set advaced options using the advancedConfiguration parameter.
    enabled: false

## Kubernetes service type
service:
  type: ClusterIP
  ## Node port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  # nodePort: 30672

  ## Set the LoadBalancerIP
  ##
  # loadBalancerIP:

  ## Node port Tls
  ##
  # nodeTlsPort: 30671

  ## Amqp port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  port: 5672

  ## Amqp Tls port
  ##
  tlsPort: 5671

  ## Dist port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  distPort: 25672

  ## RabbitMQ Manager port
  ## ref: https://github.com/bitnami/bitnami-docker-rabbitmq#environment-variables
  ##
  managerPort: 15672

  ## Service annotations
  annotations: {}
    # service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

  ## Load Balancer sources
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
  ##
  # loadBalancerSourceRanges:
  # - 10.10.10.0/24

  ## Extra ports to expose
  # extraPorts:

  ## Extra ports to be included in container spec, primarily informational
  # extraContainerPorts:

# Additional pod labels to apply
podLabels: {}

## Pod Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001
  extra: {}

persistence:
  ## this enables PVC templates that will create one per pod
  enabled: true

  ## rabbitmq data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  # storageClass: "-"
  accessMode: ReadWriteOnce

  ## Existing PersistentVolumeClaims
  ## The value is evaluated as a template
  ## So, for example, the name can depend on .Release or .Chart
  # existingClaim: ""

  # If you change this value, you might have to adjust `rabbitmq.diskFreeLimit` as well.
  size: 8Gi

  # persistence directory, maps to the rabbitmq data directory
  path: /opt/bitnami/rabbitmq/var/lib/rabbitmq

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}

networkPolicy:
  ## Enable creation of NetworkPolicy resources. Only Ingress traffic is filtered for now.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/network-policies/
  ##
  enabled: false

  ## The Policy model to apply. When set to false, only pods with the correct
  ## client label will have network access to the ports RabbitMQ is listening
  ## on. When true, RabbitMQ will accept connections from any source
  ## (with the correct destination port).
  ##
  allowExternal: true

  ## Additional NetworkPolicy Ingress "from" rules to set. Note that all rules are OR-ed.
  ##
  # additionalRules:
  #  - matchLabels:
  #    - role: frontend
  #  - matchExpressions:
  #    - key: role
  #      operator: In
  #      values:
  #        - frontend

## Replica count, set to 1 to provide a default available cluster
replicas: 1

## Pod priority
## https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
# priorityClassName: ""

## updateStrategy for RabbitMQ statefulset
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategy:
  type: RollingUpdate

## Node labels and tolerations for pod assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
nodeSelector: {}
tolerations: []
affinity: {}
podDisruptionBudget: {}
  # maxUnavailable: 1
  # minAvailable: 1
## annotations for rabbitmq pods
podAnnotations: {}

## Configure the ingress resource that allows you to access the
## Wordpress installation. Set up the URL
## ref: http://kubernetes.io/docs/user-guide/ingress/
##
ingress:
  ## Set to true to enable ingress record generation
  enabled: false

  ## The list of hostnames to be covered with this ingress record.
  ## Most likely this will be just one host, but in the event more hosts are needed, this is an array
  ## hostName: foo.bar.com
  path: /

  ## Set this to true in order to enable TLS on the ingress record
  ## A side effect of this will be that the backend wordpress service will be connected at port 443
  tls: false

  ## If TLS is set to true, you must declare what secret will store the key/certificate for TLS
  tlsSecret: myTlsSecret

  ## Ingress annotations done as key:value pairs
  ## If you're using kube-lego, you will want to add:
  ## kubernetes.io/tls-acme: true
  ##
  ## For a full list of possible ingress annotations, please see
  ## ref: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
  ##
  ## If tls is set to true, annotation ingress.kubernetes.io/secure-backends: "true" will automatically be set
  annotations: {}
  #  kubernetes.io/ingress.class: nginx
  #  kubernetes.io/tls-acme: true

## The following settings are to configure the frequency of the lifeness and readiness probes
livenessProbe:
  enabled: true
  initialDelaySeconds: 120
  timeoutSeconds: 20
  periodSeconds: 30
  failureThreshold: 6
  successThreshold: 1

readinessProbe:
  enabled: true
  initialDelaySeconds: 10
  timeoutSeconds: 20
  periodSeconds: 30
  failureThreshold: 3
  successThreshold: 1

metrics:
  enabled: false
  image:
    registry: docker.io
    repository: bitnami/rabbitmq-exporter
    tag: 0.29.0-debian-10-r28
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  ## environment variables to configure rabbitmq_exporter
  ## ref: https://github.com/kbudde/rabbitmq_exporter#configuration
  env: {}
  ## Metrics exporter port
  port: 9419
  ## RabbitMQ address to connect to (from the same Pod, usually the local loopback address).
  ## If your Kubernetes cluster does not support IPv6, you can change to `127.0.0.1` in order to force IPv4.
  ## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/#networking
  rabbitmqAddress: localhost
  ## Comma-separated list of extended scraping capabilities supported by the target RabbitMQ server
  ## ref: https://github.com/kbudde/rabbitmq_exporter#extended-rabbitmq-capabilities
  capabilities: "bert,no_sort"
  resources: {}
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9419"

  livenessProbe:
    enabled: true
    initialDelaySeconds: 15
    timeoutSeconds: 5
    periodSeconds: 30
    failureThreshold: 6
    successThreshold: 1

  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    timeoutSeconds: 5
    periodSeconds: 30
    failureThreshold: 3
    successThreshold: 1

  ## Prometheus Service Monitor
  ## ref: https://github.com/coreos/prometheus-operator
  ##      https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
  serviceMonitor:
    ## If the operator is installed in your cluster, set to true to create a Service Monitor Entry
    enabled: false
    ## Specify the namespace in which the serviceMonitor resource will be created
    # namespace: ""
    ## Specify the interval at which metrics should be scraped
    interval: 30s
    ## Specify the timeout after which the scrape is ended
    # scrapeTimeout: 30s
    ## Specify Metric Relabellings to add to the scrape endpoint
    # relabellings:
    ## Specify honorLabels parameter to add the scrape endpoint
    honorLabels: false
    ## Specify the release for ServiceMonitor. Sometimes it should be custom for prometheus operator to work
    # release: ""
    ## Used to pass Labels that are used by the Prometheus installed in your cluster to select Service Monitors to work with
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
    additionalLabels: {}

  ## Custom PrometheusRule to be defined
  ## The value is evaluated as a template, so, for example, the value can depend on .Release or .Chart
  ## ref: https://github.com/coreos/prometheus-operator#customresourcedefinitions
  prometheusRule:
    enabled: false
    additionalLabels: {}
    namespace: ""
    rules: []
      ## List of reules, used as template by Helm.
      ## These are just examples rules inspired from https://awesome-prometheus-alerts.grep.to/rules.html
      ## Please adapt them to your needs.
      ## Make sure to constraint the rules to the current rabbitmq service.
      ## Also make sure to escape what looks like helm template.
      # - alert: RabbitmqDown
      #   expr: rabbitmq_up{service="{{ template "rabbitmq.fullname" . }}"} == 0
      #   for: 5m
      #   labels:
      #     severity: error
      #   annotations:
      #     summary: Rabbitmq down (instance {{ "{{ $labels.instance }}" }})
      #     description: RabbitMQ node down

      # - alert: ClusterDown
      #   expr: |
      #     sum(rabbitmq_running{service="{{ template "rabbitmq.fullname" . }}"})
      #     < {{ .Values.replicas }}
      #   for: 5m
      #   labels:
      #     severity: error
      #   annotations:
      #     summary: Cluster down (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         Less than {{ .Values.replicas }} nodes running in RabbitMQ cluster
      #         VALUE = {{ "{{ $value }}" }}

      # - alert: ClusterPartition
      #   expr: rabbitmq_partitions{service="{{ template "rabbitmq.fullname" . }}"} > 0
      #   for: 5m
      #   labels:
      #     severity: error
      #   annotations:
      #     summary: Cluster partition (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         Cluster partition
      #         VALUE = {{ "{{ $value }}" }}

      # - alert: OutOfMemory
      #   expr: |
      #     rabbitmq_node_mem_used{service="{{ template "rabbitmq.fullname" . }}"}
      #     / rabbitmq_node_mem_limit{service="{{ template "rabbitmq.fullname" . }}"}
      #     * 100 > 90
      #   for: 5m
      #   labels:
      #     severity: warning
      #   annotations:
      #     summary: Out of memory (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         Memory available for RabbmitMQ is low (< 10%)\n  VALUE = {{ "{{ $value }}" }}
      #         LABELS: {{ "{{ $labels }}" }}

      # - alert: TooManyConnections
      #   expr: rabbitmq_connectionsTotal{service="{{ template "rabbitmq.fullname" . }}"} > 1000
      #   for: 5m
      #   labels:
      #     severity: warning
      #   annotations:
      #     summary: Too many connections (instance {{ "{{ $labels.instance }}" }})
      #     description: |
      #         RabbitMQ instance has too many connections (> 1000)
      #         VALUE = {{ "{{ $value }}" }}\n  LABELS: {{ "{{ $labels }}" }}

##
## Init containers parameters:
## volumePermissions: Change the owner of the persist volume mountpoint to RunAsUser:fsGroup
##
volumePermissions:
  enabled: false
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: buster
    pullPolicy: Always
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  resources: {}

## forceBoot: executes 'rabbitmqctl force_boot' to force boot cluster shut down unexpectedly in an
## unknown order.
## ref: https://www.rabbitmq.com/rabbitmqctl.8.html#force_boot
##
forceBoot:
  enabled: false

## Optionally specify extra secrets to be created by the chart.
## This can be useful when combined with load_definitions to automatically create the secret containing the definitions to be loaded.
##
extraSecrets: {}
  # load-definition:
  #   load_definition.json: |
  #     {
  #       ...
  #     }

The values.yaml file is the configuration information publicly disclosed by the helm chart, which can be modified as needed. Because we are using Alibaba Cloud, we use alicloud-disk-ssd for storage. Note that the minimum required ssd for alicloud-disk-ssd is 20G:

storageClass: "alicloud-disk-ssd"
size: 20Gi

Next, we create the rabbit namespace

kubectl create namespace rabbit

Installation in development and test environment

Because we need to use ip:port to access the development environment, we need to configure the service.
The final complete values.yaml file is as follows:

values.yaml

service:
  type: NodePort
persistence:
  storageClass: "alicloud-disk-ssd"
  size: 20Gi

Now let's install rabbitmq, run it with the following command:

# 创建rabbitmq集群
helm install -f values.yaml test-rabbitmq bitnami/rabbitmq --namespace rabbit

The following is the output during installation:

output start

$ helm install -f test-values.yaml test-rabbitmq bitnami/rabbitmq --namespace rabbit
NAME: test-rabbitmq
LAST DEPLOYED: Thu Jul  8 10:07:50 2021
NAMESPACE: rabbit
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Credentials:
    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"
    echo "ErLang Cookie : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)"

Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases.

RabbitMQ can be accessed within the cluster on port  at test-rabbitmq.rabbit.svc.

To access for outside the cluster, perform the following steps:

Obtain the NodePort IP and ports:

    export NODE_IP=$(kubectl get nodes --namespace rabbit -o jsonpath="{.items[0].status.addresses[0].address}")
    export NODE_PORT_AMQP=$(kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq)
    export NODE_PORT_STATS=$(kubectl get --namespace rabbit -o jsonpath="{.spec.ports[3].nodePort}" services test-rabbitmq)

To Access the RabbitMQ AMQP port:

    echo "URL : amqp://$NODE_IP:$NODE_PORT_AMQP/"

To Access the RabbitMQ Management interface:

    echo "URL : http://$NODE_IP:$NODE_PORT_STATS/"
Output end

Check all the rabbitmq resources under the rabbit namespace to see if they are successfully created!

$ kubectl get all -n rabbit
NAME                  READY   STATUS    RESTARTS   AGE
pod/test-rabbitmq-0   1/1     Running   0          3h56m

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                 AGE
service/test-rabbitmq            ClusterIP   172.21.15.81   <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m
service/test-rabbitmq-headless   ClusterIP   None           <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m

NAME                             READY   AGE
statefulset.apps/test-rabbitmq   1/1     3h56m

After waiting for a while, we will find that pod, svc, pvc, pv, and statefulset are all created. How do we access the Rabbitmq management interface? From the output content, we find:

  • NODE_IP
    Command: kubectl get nodes --namespace rabbit -o jsonpath="{.items[0].status.addresses[0].address}"
  • NODE_PORT_AMQP
    Command: kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq
  • NODE_PORT_STATS
    Command: kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq

The above Node_ip is actually the real ip of the k8s node (see if it is accessible from the public network, if not, you can apply for Alibaba Cloud EIP to bind to the public network to access it), such as: 192.168.0.1, and NODE_PORT_AMQP is corresponding to the cluster 5672, such as 31020, NODE_PORT_STATS is the NodePort port corresponding to 15672, such as 31010
In this way, we can access the Rabbitmq management interface and enter: http://192.168.0.1:31010 in the browser to access:

image.png

In this way, springboot can access Rabbitmq through the ip address,

spring:
  rabbitmq:
    host: 192.168.0.1
    port: 5672
    username: user
    password:

Installation in a pre-production production environment

Because we need to access through the domain name for development, we need to enable ingress and domain name configuration:

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
  hostName: rabbitmq.demo.com

The last is to access via https, we have to enable tls:

tls: true
tlsSecret: tls-secret-name

Let's pay attention here, because I use the cert-manager.io/cluster-issuer annotation (I have already generated it in k8s), so I can directly generate the tls certificate, which is very convenient. If you are interested, you can see and use cert-manager applies for a free HTTPS certificate

annotations:
    cert-manager.io/cluster-issuer: your-cert-manager-name

The final complete values.yaml file is as follows:

values.yaml

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: your-cert-manager-name
    nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
  hostName: rabbitmq.demo.com
  tls: true
  tlsSecret: tls-secret-name
persistence:
  storageClass: "alicloud-disk-ssd"
  size: 20Gi

Now let's install rabbitmq, run it with the following command:

# 创建rabbitmq集群
helm install -f values.yaml test-rabbitmq bitnami/rabbitmq --namespace rabbit

The following is the output during installation:

output start

NAME: test-rabbitmq
LAST DEPLOYED: Thu Jul  8 19:06:58 2021
NAMESPACE: rabbit
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Credentials:
    echo "Username      : user"
    echo "Password      : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"
    echo "ErLang Cookie : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)"

Note that the credentials are saved in persistent volume claims and will not be changed upon upgrade or reinstallation unless the persistent volume claim has been deleted. If this is not the first installation of this chart, the credentials may not be valid.
This is applicable when no passwords are set and therefore the random password is autogenerated. In case of using a fixed password, you should specify it when upgrading.
More information about the credentials may be found at https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/#credential-errors-while-upgrading-chart-releases.

RabbitMQ can be accessed within the cluster on port  at test-rabbitmq.rabbit.svc.

To access for outside the cluster, perform the following steps:

To Access the RabbitMQ AMQP port:

1. Create a port-forward to the AMQP port:

    kubectl port-forward --namespace rabbit svc/test-rabbitmq 5672:5672 &
    echo "URL : amqp://127.0.0.1:5672/"

2. Access RabbitMQ using using the obtained URL.

To Access the RabbitMQ Management interface:

1. Get the RabbitMQ Management URL and associate its hostname to your cluster external IP:

   export CLUSTER_IP=$(minikube ip) # On Minikube. Use: `kubectl cluster-info` on others K8s clusters
   echo "RabbitMQ Management: http://rabbitmq.demo.com/"
   echo "$CLUSTER_IP  rabbitmq.testapi.seaurl.com" | sudo tee -a /etc/hosts

2. Open a browser and access RabbitMQ Management using the obtained URL.
End of output

Check all the rabbitmq resources under the rabbit namespace to see if they are successfully created!

$ kubectl get all -n rabbit
NAME                  READY   STATUS    RESTARTS   AGE
pod/test-rabbitmq-0   1/1     Running   0          3h56m

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                 AGE
service/test-rabbitmq            ClusterIP   172.21.15.81   <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m
service/test-rabbitmq-headless   ClusterIP   None           <none>        4369/TCP,5672/TCP,25672/TCP,15672/TCP   3h56m

NAME                             READY   AGE
statefulset.apps/test-rabbitmq   1/1     3h56m

After waiting for a while, we will find that pod, svc, ingress, pvc, pv, and statefulset are all created. How do we access the Rabbitmq management interface? From the output, we find that http://rabbitmq.demo.com/
Address, in this way, we visit the Rabbitmq management interface and enter in the browser: http://rabbitmq.demo.com/ to complete the visit:

image.png

In this way, springboot can access Rabbitmq through the name parsed by k8s intranet dns, how to parse it can be test this article

spring:
  rabbitmq:
    host: test-rabbitmq-headless.rabbit.svc.cluster.local
    port: 5672
    username: user
    password:

image.png

problem

1. Visit rabbitmq report: 503
If you configure the domain name path such as: demo.com/rabbitmq , such a domain name, then you must configure it as follows to access it correctly. In addition: recommended to use the first-level or second-level or third-level domain name as the hostName. Do not use path:/rabbitmq One form refer to this article :

rabbitmq:
  extraConfiguration: |-
    management.path_prefix = /rabbitmq/
ingress:
...
  hostName: demo.com
  path: /rabbitmq/
...

2, kubectl describe svc your-service-name -n rabbit found that the service endpoint is empty

3、running "VolumeBinding" filter plugin for pod "test-rabbitmq-0": pod has unbound immediate PersistentVolumeClaims

  • Reason: To set the storageClass and the minimum requirement of Alibaba Cloud Disk is 20Gi
  • solve:

    persistence:
    storageClass: "alicloud-disk-ssd"
    size: 20Gi

4、PersistentVolumeClaim "data-test-rabbitmq-0" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims

  • Reason: PVC container does not support online modification
  • Solution: delete pvc and recreate

5、Warning ProvisioningFailed 6s (x4 over 14m) diskplugin.csi.alibabacloud.com_iZbp1d2cbgi4jt9oty4m9iZ_3408e051-98e8-4295-8a21-7f1af0807958 (combined from similar events): failed to provision volume with StorageClass "alicloud-disk-ssd": rpc error: code = Internal desc = SDK.ServerError
ErrorCode: InvalidAccountStatus.NotEnoughBalance

  • Reason: Alibaba Cloud account balance must be at least 100 yuan to create an SSD cloud disk
  • Solution: Alibaba Cloud account recharge

6. Log in to rabbitmq and report 401
Reason: wrong password
Solution: Run the following command to get the password

echo "Password      : $(kubectl get secret --namespace rabbit test-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 --decode)"

7. Springcloud microservice access port 5672 connection report: time out
Reason: spring.rabbitmq.host address should not be your ingress extranet domain name: rabbitmq.demo.com, but should be your intranet cluster dns resolved address
Solution: Use the following command to obtain the internal network resolution address of test-rabbitmq, and then assign it to spring.rabbitmq.host to reconnect.

# 如果 pod 和服务的命名空间不相同,则 DNS 查询必须指定服务所在的命名空间。
# test-rabbitmq-headless.rabbit命名空间是rabbit
# 如果没有指定命名空间,可以使用kubectl exec -i -t dnsutils -- nslookup test-rabbitmq-headless即可
$ kubectl exec -i -t dnsutils -- nslookup test-rabbitmq-headless.rabbit
Server:        172.21.0.10
Address:    172.21.0.10#53

Name:    test-rabbitmq-headless.rabbit.svc.cluster.local
Address: 172.20.0.124

application.yml

spring.rabbitmq.host=test-rabbitmq-headless.rabbit.svc.cluster.local

8. The NodePort port corresponding to 5672 obtained in the development and test environment is incorrect
Reason: The 5672 port obtained by the following command is incorrect

kubectl get --namespace rabbit -o jsonpath="{.spec.ports[1].nodePort}" services test-rabbitmq

Solution: Use the following command to view and obtain the NodePort port 31972 corresponding to 5672

$ kubectl get svc test-rabbitmq -n rabbit
NAME            TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                         AGE
test-rabbitmq   NodePort   172.21.14.17   <none>        5672:31972/TCP,4369:31459/TCP,25672:31475/TCP,15672:31696/TCP   14m

to sum up

1. Kubectl needs to configure the connection configuration information of k8s before it can be used, and helm gets the connection information by default, which is in ~/.kube/config by default
2. Helm3 has removed the helm tillers and helm init commands, so don't discuss helm2.
3. If an error occurs during or after the installation, you can delete it from the application->helm release list under the Aliyun ack console.
4. There is no need to configure rabbitmq-plugins enable rabbitmq_management , because helm starts by default after installing rabbitmq.
5. If you want to debug locally, you can use the following method

To Access the RabbitMQ AMQP port:

    kubectl port-forward --namespace rabbit svc/test-rabbitmq 5672:5672
    echo "URL : amqp://127.0.0.1:5672/"

To Access the RabbitMQ Management interface:

    kubectl port-forward --namespace rabbit svc/test-rabbitmq 15672:15672
    echo "URL : http://127.0.0.1:15672/"

6, rabbitmq 5672 port only supports amqp: protocol, which means that ingress cannot be exposed, and ingress can only expose http and https, so we set the service type: NodePort in the development test environment for this reason! ! !

Quote

Helm deploys RabbitMQ cluster _
helm document
How does kubectl exec work?
Helm tutorial

阅读 1.5k

全栈工程师进阶
日常学习总结与分享,包括:前端、后台与运维,讲解的知识点包括:javascript、vuejs、reactjs、springb...

Awbeci

2.7k 声望
163 粉丝
0 条评论
你知道吗?

Awbeci

2.7k 声望
163 粉丝
宣传栏