EMQX 5.0 New Gateway Framework: Easy Access to Multiple IoT Protocols

Introduction: Gateway Framework for Unified Management

As a large-scale distributed IoT MQTT message server, EMQX not only fully supports MQTT 3.x and 5.0, but also supports STOMP, MQTT-SN , LwM2M/ CoAP , JT/T 808 and other mainstream protocol access. It not only provides a wide range of connectivity capabilities to handle IoT devices suitable for various scenarios, but also provides a unified access platform and management interface for back-end IoT management services to reduce the cost of adaptation between heterogeneous protocols.

In the newly released EMQX 5.0 , we reconstructed the underlying architecture of multi-protocol access, unified the configuration format and management interface, and provided a brand-new extended gateway framework. At the same time, the implementation of various gateways is standardized, making the definition of each gateway function clearer.

This article will explain in detail the use of the new gateway framework and functions of EMQX 5.0, to help readers make better use of EMQX's powerful multi-protocol access support capabilities to connect various protocol devices and meet the data access needs of more IoT scenarios.

Overview of EMQX Gateway Framework

Due to the variety of device access protocols, each protocol has different definitions for connection login, message transmission, etc., which usually requires the server to deploy access services of various protocols, resulting in a sharp increase in software and maintenance costs. Since its release, EMQX has provided multi-protocol access support to shield the heterogeneity of access layer protocols to reduce user development and operation costs.

Before version 5.0, EMQX's multi-protocol support was realized by publishing access plug-ins for various protocols. These independent plug-ins lacked unified definitions and standards, and were relatively difficult to use.

In order to bring more convenient and easy-to-use multi-protocol support to users, EMQX 5.0 reconstructs the entire multi-protocol access architecture. All non-MQTT protocol accesses are collectively referred to as gateways. All gateways are supported by a unified framework that provides common operations, including:

• Unified user interface: The framework provides a unified configuration file, HTTP API and command line interface. Taking the listener parameter configuration as an example, different protocol plugins in version 4.x expose different parameters to the listener, but in version 5.0, the style of these parameters will be unified.
• Unified statistics and monitoring indicators: Provides gateway and client-level statistical indicators, such as the number of bytes sent and received, messages, etc.
• Independent connection and session management: Each gateway has its own client management page, and different gateways are allowed to use the same Client ID, instead of being mixed in the MQTT client list for management as in the 4.x version.
• Independent client authentication: supports configuring independent authentication for each gateway, no longer mixed with MQTT client authentication like 4.x.
• Easy to extend and clear specifications: The framework abstracts a standard set of concepts and interfaces to make it easier to customize gateways.

The implementation within each gateway is similar to the previous:

• Listener: Each gateway can start multiple listeners to accept network requests from clients. The listener types support TCP, SSL, UDP, and DTLS. Each type of gateway supports different types of listeners.
• Packet parsing: Each gateway has its own packet parsing module, which is responsible for processing packets of this protocol.
• Connection/Session: Responsible for creating connections, sessions, and handling various behaviors defined in the protocol, such as login authentication, messaging, etc.
• Message model conversion: Responsible for handling the compatibility of this gateway with the MQTT PUB/SUB message model. For example, convert messages in LwM2M to messages with topic and QoS in EMQX.

Gateway General Code of Conduct

In addition to the above architectural reconstruction, the gateway of EMQX 5.0 also unifies common behaviors such as access authentication and message sending and receiving.

Access Authentication: Client Information

The gateway uses the client information uniformly for authentication. The client information is created by the gateway when processing the client access, where:

• No matter what kind of gateway, its client information includes common fields, such as Client ID, Username, Password, etc. (even if the protocol does not have a definition of this field, the gateway will set a suitable default value for it) It also includes Peername, ProtoName, Peercert Wait
• Each gateway also has its specific client information, such as LwM2M has Endpoint Name and Life Time and so on.

Therefore, when performing client authentication, both such generic client fields and specific fields can be passed as parameters to the authenticator to perform validation.

Messaging: PUB/SUB message model conversion

In order to adapt to the PUB/SUB message model of MQTT, each type of gateway must be compatible with this message model in order to communicate with each other. For PUB/SUB type protocol gateways, such as MQTT-SN and STOMP, the concepts of topic and message payload are usually defined, then:

• Directly use client-specified subject and message content
• Choose an appropriate value as the QoS of the message.

But for non-PUB/SUB type protocols, it lacks definitions of concepts such as topics, publish, subscribe, etc., then:

• It needs to be specified with a message subject. For example, LwM2M gateway, users can configure the subject of each type of message.
• The format for which the message content needs to be designed. Each type of gateway may use a different message format.

Detailed Explanation of EMQX 5.0 Gateway Framework

Client authentication

Each type of device in EMQX 4.x uses the same authentication chain as MQTT. This coupling leads to the need to consider the heterogeneous situation of each gateway when configuring the authenticator:

In EMQX 5.0, the gateway framework allows to configure its own authenticator for each type of gateway:

Change this picture: you only need to compare MQTT and LwM2M, remove MQTT-SN and others, and change the top and bottom of the previous authenticator to the left and right.

message model conversion

Message model transformation does not apply to gateways that have concepts such as PUB/SUB defined. For example, the MQTT-SN protocol has defined the behavior of publish/subscribe, then the MQTT-SN gateway will:

• Publish the PUBLISH packet in the protocol as a message, and its subject and QoS are specified by this packet.
• The SUBSCRIBE message in the protocol is used as a subscription operation, and its topic and QoS are specified by this message.
• Use the UNSUBSCRIBE message of the protocol as the unsubscribe operation, and its topic is specified by this message.

The message model transformation also does not work for gateways that have a concept close to the PUB/SUB model. For example, if the STOMP protocol is fully compatible with this message model, the STOMP gateway will:

• Publish the SEND message in the protocol as a message. Its subject is the destination field in the SEND message, the message content is the message body content of the SEND message, and the QoS is fixed to 0.
• Use the SUBSCRIBE message in the protocol as a subscription request. The subject is the destination field in the SUBSCRIBE packet, and the QoS is fixed to 0. And supports wildcards defined in the MQTT protocol.
• Use the UNSUBSCRIBE message in the protocol as an unsubscribe request. Its subject is the destination field in the UNSUBSCRIBE message.

Message model transformation is only applicable to gateways that do not define concepts such as PUB/SUB. For example, for the LwM2M protocol, you need to add some configuration to it to specify the topic format used, and the gateway will have built-in rules to organize the format of the message content:

 gateway.lwm2m {

  mountpoint = "lwm2m/${endpoint_name}/"

  translators {  
    // 下行命令主题。
    // 对于每个成功上线的新 LwM2M 客户端，网关会创建一个订阅关系来接收下行消息并将其发送给客户端
    command {
      topic = "dn/#"
      qos = 0
    }                                         

    // 用于发布来自 LwM2M 客户端的注册事件的主题                   
    register {
      topic = "up/register"
      qos = 0
    }
    ...                                   
  } 
}

Then, if a client whose endpoint_name is epn1 online:

• The gateway subscribes to the lwm2m/epn1/dn/# topic for its proxy, expecting to receive downstream control messages
• The gateway will publish the client's REGISTER message to the topic lwm2m/epn1/up/register . Its message format is determined by the data conversion rules defined by the LwM2M gateway. For example, the format of the REGISTER message is:

 { "msgType": "register",                                                
  "data": {                                                             
    "ep": "epn1",                                            
    "lt": 6400,                                                
    "sms": "sms_no_example",                                                 
    "lwm2m": "1.2",          
    "objectList": ["1/0", "3/0", "19/0"]                                       
  }                                                                     
}

Publish-Subscribe Authorization

There is no independent subject authorization management in the gateway, they are all centralized in AuthZ. Reference: Authorization

Note: Gateways using PUB/SUB model transformation do not need to set topic configuration permissions on them, because topic rules for such clients are mandatory.

hook support

EMQX relies on hooks to achieve the expansion of various functions, such as online and offline messages, and the triggering of the rule engine. Therefore, gateways must publish critical events to hooks for compatibility with other EMQX functions.

In v4.x, there is no specification for the support of each hook. In v5.0, we summarize it. The following are the hooks that must be supported:

E.g:

• The LwM2M gateway supports the client.connected hook, so the rule engine can get the online event of each LwM2M device through $event/client_connected .
• LwM2M gateway supports client.authenticate hook, so ExHook can handle LwM2M client authentication by mounting this hook.

custom authentication

The gateway, like the MQTT client , also distributes authentication requests based on the authentication chain until an authenticator, plugin or ExHook on the chain returns an allow/deny:

Therefore, support for authentication can also be extended by custom authentication plugins or using ExHook . The semantics of each authenticator can be abbreviated as:

 fun authenticate(ClientInfo, LastAuthResult)  // 入参为：客户端信息、和上次认证器的执行结果
    -> {stop, NewAuthResult}                  // 返回情况1：终止链执行，并返回新的认证结果
     | ignore                                 // 返回情况2：忽略，并继续执行链上的下一个认证器

Note: All protocols will publish authentication requests to the authentication chain, so it is necessary to distinguish which type of gateway and listener the client is from by fields such as protocol and listener_id

user interface

The gateway framework provides a unified user-level interface for all gateways. For example, the gateway's HTTP API can be used to achieve:

• Enable, stop, and update configuration of a gateway, etc.
• Enable, disable, update the authenticator of a gateway, etc.
• Add, delete, update the listener of a gateway, etc.
• Query, kick out the client of a gateway, or add or unsubscribe for a client, etc.

This article only provides some simple examples. For details, please refer to the official website documentation: Gateway Configuration , Gateway HTTP API

Example 1, enabling a STOMP gateway via a configuration file:

 gateway.stomp {

  mountpoint = "stomp/"

  listeners.tcp.default {
    bind = 61613
    acceptors = 16
    max_connections = 1024000
    max_conn_rate = 1000
  }
}

Example 2, enabling an MQTT-SN gateway via the HTTP API:

 curl -X 'POST' 'http://127.0.0.1:18083/api/v5/gateway' \
  -u admin:public \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "mqttsn",
  "enable": true,
  "gateway_id": 1,
  "mountpoint": "mqttsn/",
  "listeners": [
    {
      "type": "udp",
      "bind": "1884",
      "name": "default",
      "max_conn_rate": 1000,
      "max_connections": 1024000
    }
  ]
}'

future outlook

The new upgrade of the gateway framework brings a lot of convenience for users to use EMQX 5.0 for multi-protocol device access. In the future, we will continue to optimize this function in the following aspects:

• Monitoring information supports output to Prometheus and StatsD for increased observability.
• Standardized tests for individual protocol implementations.
• Improve the personalized management interface and client information. For example, it operates on the resource model of the LwM2M device.
• UDP protocol access such as LwM2M needs to support session identification under NAT network. Once the device sleeps, the address and port to the server will change after the NAT translation will cause the LwM2M device to wake up. Therefore, some mechanism should be designed to identify this change.
• Implement a lighter design for ExProto to reduce the difficulty of using gRPC and improve operating efficiency.

Epilogue

The access and unified management of multiple protocols are realized through the new gateway framework, which further improves the usability of EMQX. Combined with powerful data integration, safe and reliable authentication and authorization, and the ability to scale horizontally at 100 million levels, IoT users in various industries can use EMQX in various business scenarios to efficiently connect and move IoT real-time data. with processing.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/emqx-connects-multiple-iot-protocols