How EMQX Operator quickly creates an elastically scalable MQTT cluster

Introduction: Embracing Cloud-Native EMQX 5.0

The cloud native concept has gradually penetrated into the application development of key businesses of various enterprises. For a cloud-native application, horizontal scaling and elastic clustering are important features it should have.

As a large-scale distributed open source IoT MQTT message server that actively embraces cloud native, the latest version 5.0 of EMQX adopts a new back-end storage architecture, the Mria database, and reconstructs the data replication logic, adding the role of Replicant node, enabling users to It can get rid of the limitation of stateful nodes, perform more elastic horizontal expansion of EMQX clusters, and create IoT applications that are more in line with the cloud-native concept.

Users can use the EMQX Kubernetes Operator , a management tool released by EMQ, to use the Replicant node feature of EMQX 5.0 to deploy stateless nodes on Kubernetes through Deployment resources, and quickly create and manage EMQX clusters that can carry large-scale MQTT connections and message throughput.

This article will help readers to further master how to quickly create, deploy and automate the management of elastically scalable EMQX clusters through a detailed explanation of the core features and application operations of EMQX Kubernetes Operator, make full use of EMQX 5.0's cloud-native support features, and embrace cloud-native.

What is EMQX Kubernetes Operator

I believe everyone is no stranger to Kubernetes (K8s). It is a widely used open source platform for automating the deployment, scaling and management of containerized applications.

On Kubernetes, Operators are software extensions to the Kubernetes API that use custom resource definitions (CRDs) to provide an application-specific API. Operators follow basic Kubernetes principles, such as using Controllers ( Controller loops ) to regulate the state of the system.

The Operator pattern combines custom resources (CRDs) and custom controllers to encode the domain knowledge of the application as extensions to the Kubernetes API that automate common coordination tasks.

The EMQX Kubernetes Operator is an EMQX-specific controller that allows DevOps to manage the lifecycle of EMQX cluster deployments in Kubernetes. The EMQX Kubernetes Operator runs as a custom controller on Kubernetes and communicates with the Kubernetes API server (kube-apiserver) to convert high-level descriptions into normal Kubernetes resources to maintain the desired application state.

In short, EMQX Kubernetes Operator can help users to quickly create and manage EMQX clusters in the Kubernetes environment, which not only greatly simplifies the deployment and management process, but also reduces the professional skills required for management and configuration.

After years of iteration on EMQX, we have also accumulated rich experience in EMQX deployment, tuning, and operation and maintenance. The EMQX Kubernetes Operator is an example of how we translate these experiences into code. It turns deployment and management work into a low-cost, standardized, and repeatable capability, helping users efficiently achieve cluster expansion, seamless upgrades, fault handling, and unified monitoring.

EMQX Kubernetes Operator VS EMQX Helm Chart

Writing YAML files by hand is not fun for most people, and writing custom YAML by hand every time you need to deploy an application or modify configuration settings in a Kubernetes cluster is a complex and error-prone task.

Helm Charts and Operators in Kubernetes solve this dilemma. These two handy tools for automating tasks provide administrators with an easy way to deploy applications or configurations into Kubernetes clusters. That way, they can take better advantage of Kubernetes. But just because they do similar things doesn't mean they're completely interchangeable.

In addition to Operator, EMQX also provides a Helm Chart deployment method on Kubernetes. Users can choose a more suitable deployment method according to their needs:

• EMQX Helm Chart

  Helm is the package management system for Kubernetes. Using a packaging format called Helm Chart, someone can package an application (such as Apache HTTP) into a format that anyone else can deploy to a Kubernetes cluster with few or no manual changes to the YAML file .

  If you are familiar with package management in a Linux environment, Helm charts should be easy to understand. They are similar to Debian or RPM packages, and Helm itself is similar to apt or dnf. Just as you can on Ubuntu apt-get install [some package] , you can use Kubernetes helm install [some package] to get applications up and running quickly.

  EMQX has provided EMQX Helm Chart since version 4.0. Through EMQX Helm Chart, users can quickly deploy an EMQX cluster on Kubernetes and complete the initialization operation. The use of Helm Chart is simple enough and suitable for users who are new to EMQX for deployment and early adopters.

  However, although Helm Chart is easy to use, it can only provide the most basic deployment capabilities. For the operation and maintenance of stateful clusters such as EMQX, Kubernetes Operator is a better choice.

• EMQX Kubernetes Operator

  As mentioned above, through Kubernetes Custom Resource (CRD), users can use declarative API to describe EMQX clusters, and the EMQX Kubernetes Operator will continuously schedule Kubernetes resources, so that EMQX clusters are ultimately consistent with what the user declares. A large number of operation and maintenance and update operations are automatically completed by the EMQX Kubernetes Operator, and users do not need to care.

  EMQX Kubernetes Operator allows us to manage and operate EMQX clusters more flexibly, freeing us from complicated and error-prone configuration modification work, thus saving a lot of costs.

  EMQX Helm Chart and EMQX Kubernetes Operator do not compete with each other, but complement each other. EMQX provides these two deployment methods at the same time, so that users can choose the most suitable one for them in different scenarios.

EMQX Kubernetes Operator upgraded with EMQX 5.0

With the update of EMQX 5.0 in the new HOCON format configuration, Replicant role node, etc., we also provide a shortcut for users to easily deploy and operate EMQX in the complex environment of Kubernetes - the upcoming EMQX Kubernetes Operator 2.0 can perfectly support EMQX The deployment management of 5.0 has been optimized and upgraded in terms of cluster strategy and configuration format:

• Brand new cluster strategy

  EMQX Kubernetes Operator 2.0 still uses Statefulset resources to deploy EMQX Core nodes, which is consistent with the previous features. Users can still persist EMQX business data through Kubernetes resources such as StoreClass, and ensure the orderliness of nodes.

  Unlike before, EMQX Kubernetes Operator 2.0 utilizes the Deployment resource to deploy EMQX Replicant nodes. Compared with Core nodes, Replicant nodes are independent of each other and only make requests to Core nodes to form a cluster. Replicant nodes are not bound to any persistent resources, which means they can be destroyed and rebuilt at any time. Users can quickly scale the number of Replicant nodes by modifying EMQX custom resources to handle their own business more flexibly.

  

  EMQX Kubernetes Operator 2.0 will route all management requests (such as Dashboard, API) to Core nodes, and route all business requests to Replicant nodes to improve cluster stability.

• New configuration format

  In the previous version, the EMQX Kubernetes Operator passed the configuration to EMQX through environment variables, which means that if the configuration is modified, the Pod will be restarted, and users are required to master the conversion rules of EMQX configuration and environment variables. Very friendly. EMQX Kubernetes Operator 2.0 will take advantage of EMQX's new HOCON configuration and Dashboard's hot configuration function, allowing users to write native EMQX configuration into EMQX custom resources, and encourage users to modify EMQX through EMQX Dashboard's hot configuration function when EMQX is running Configuration. These configurations will take effect in the entire cluster, and the configuration will be synchronized when a new node joins the cluster to ensure the consistency of all nodes.

• Brand new upgrade management

  In EMQX 5.0, cluster upgrade/downgrade has become more complicated because of the introduction of different cluster roles. Users need to upgrade EMQX Core nodes first, wait for all Core nodes to be upgraded and restore the cluster, and then upgrade Replicant nodes.

  To simplify this process, EMQX Kubernetes Operator 2.0 provides upgrade management capabilities. When users want to upgrade/downgrade EMQX, they only need to directly modify the Image in the EMQX custom resource, and the EMQX Kubernetes Operator will automatically complete all the work.

Quickly deploy EMQX 5.0 with EMQX Kubernetes Operator

With the EMQX Kubernetes Operator, an EMQX cluster can be deployed with just a few lines of YAML.

 $ cat << "EOF" | kubectl apply -f -                
  apiVersion: apps.emqx.io/v2alpha1
  kind: EMQX
  metadata:
    name: emqx 
  spec:                      
    emqxTemplate:   
      image: emqx/emqx:5.0.6
EOF
emqx.apps.emqx.io/emqx applied

EMQX Kubernetes Operator deploys 3 Core nodes and 3 Replicant nodes by default. Users can scale the number of nodes by modifying EMQX custom resources.

 $ kubectl get pods 
NAME                              READY   STATUS    RESTARTS        AGE
emqx-core-0                       1/1     Running   0               75s
emqx-core-1                       1/1     Running   0               75s
emqx-core-2                       1/1     Running   0               75s
emqx-replicant-6c8b4fccfb-bkk4s   1/1     Running   0               75s
emqx-replicant-6c8b4fccfb-kmg9j   1/1     Running   0               75s
emqx-replicant-6c8b4fccfb-zc929   1/1     Running   0               75s
$ kubectl get emqx emqx -o json | jq ".status.emqxNodes"
[
  {
    "node": "emqx@172.17.0.11",
    "node_status": "running",
    "otp_release": "24.2.1-1/12.2.1",
    "role": "replicant",
    "version": "5.0.6"
  },
  {
    "node": "emqx@172.17.0.12",
    "node_status": "running",
    "otp_release": "24.2.1-1/12.2.1",
    "role": "replicant",
    "version": "5.0.6"
  },
  {
    "node": "emqx@172.17.0.13",
    "node_status": "running",
    "otp_release": "24.2.1-1/12.2.1",
    "role": "replicant",
    "version": "5.0.6"
  },
  {
    "node": "[e-mail]",
    "node_status": "running",
    "otp_release": "24.2.1-1/12.2.1",
    "role": "core",
    "version": "5.0.6"
  },
  {
    "node": "[e-mail]",
    "node_status": "running",
    "otp_release": "24.2.1-1/12.2.1",
    "role": "core",
    "version": "5.0.6"
  },
  {
    "node": "[e-mail]",
    "node_status": "running",
    "otp_release": "24.2.1-1/12.2.1",
    "role": "core",
    "version": "5.0.6"
  }
]

Epilogue

In addition to stronger horizontal expansion capabilities, EMQX 5.0 also provides clearer and comprehensive data monitoring and management capabilities through the newly revised Dashboard , improving observability. In addition, the implementation of MQTT over QUIC support will enable the MQTT connection based on the QUIC protocol to switch to another Pod unconsciously when the Pod is scheduled, thereby further improving the availability of the cluster. These will enable users to build more cloud-native applications with EMQX 5.0.

EMQX Kubernetes Operator provides a more convenient way for users to create and manage EMQX clusters, helping users to experience the cloud-native features of EMQX 5.0 more easily.

In the future, EMQ will continue to work in the cloud-native direction, and evolve EMQX into a flexible and stateless MQTT Broker. At the same time, it will cooperate with EMQ edge computing products such as eKuiper and Neuron to further explore the implementation of distributed cloud-native.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/create-a-scalable-mqtt-cluster-with-emqx-operator