NanoMQ Newsletter 2022-06｜Rules engine officially released & NanoSDK supports MQTT over QUIC

NanoMQ continues to be updated steadily, and 0.9.0 will be officially released in early July. This release brings you 2 important feature updates: the rules engine and NanoSDK which supports QUIC. At the same time, offline data cache configuration has been added, and various performance optimizations and bug fixes are also in progress.

Lightweight and easy-to-use embedded rule engine

The rule engine is a favorite feature of EMQX by the majority of users. NanoMQ has also launched the same feature according to user needs. Now users can easily write SQL rules to process messages and then republish them or store them in the database for persistence.

NanoMQ's rule engine consists of a unified messaging interface and various target plugins, each of which has its own configuration file. Currently only the SQLite plugin is available as a data persistence option, and more optional plugins will be added in the future.

To use the rule engine, you need to enable the SQLite plugin function in the global configuration /etc/nanomq_rule.conf:

 ## Rule engine option, when persistence with 
## rule engine, this option is must be ON.
## 
## Value: ON | OFF
rule_option=ON

## Rule engine plugins option
## Choose a plugin to enable
## 
## Value: enable/disable
rule_option.sqlite=enable

## Rule engine option database config path
## Rule engine plugin config path, default is exec path.
## 
## Value: File nanomq_rule_sqlite.conf
rule_option.sqlite.conf.path=/etc/nanomq_rule_sqlite.conf

Then configure the corresponding SQL statements and database parameters in the corresponding plugin configuration file /etc/nanomq_rule_sqlite.conf:

 ## Rule engine option SQLite3 database path
## Rule engine db path, default is exec path.
## 
## Value: File
rule.sqlite.path=/tmp/sqlite_rule.db

## Rule engine option SQLite3 database table name
## Rule engine db table name.
## 
## Value: String
rule.sqlite.table=broker

## Rule engine option sql
## Rule engine sql clause.
## 
## Value: String
SELECT * FROM "abc" WHERE payload.x.y = 10

With this configuration, NanoMQ will use the SQL in the configuration to process all Publish messages and persist them to the corresponding SQLite database table.

For example, at this point we publish two different messages to the abc topic:

 msg1: "{"x": {"y": 10}, "z": "str1", "a": 1111}
msg2: "{"x": {"y": 11}, "z": "str1", "a": 1111}

Then you can see in the SQLite database:

 sqlite> SELECT * FROM Broker;
RowId|Qos|Id|Topic|Clientid|Username|Password|Timestamp|Payload
1|1|1|abc|nanomq-57a0d6ab|abc|(null)|1656066147|{"x": {"y": 10}, "z": "str1", "a": 1111}
2|1|1|abc|nanomq-9c1ca526|abc|(null)|1656066147|{"x": {"y": 10}, "z": "str1", "a": 1111}
3|1|1|abc|nanomq-7ff24b6f|abc|(null)|1656066147|{"x": {"y": 10}, "z": "str1", "a": 1111}
4|1|1|abc|nanomq-83e7ff63|abc|(null)|1656066147|{"x": {"y": 10}, "z": "str1", "a": 1111}

It can be seen that only the message msg1 that meets the rules is persisted to the database. The rules engine currently supports standard JSON parsing and common SQL statements and symbols. For details, please refer to the documentation.

At present, the running order of NanoMQ's rule engine is serially executed after processing MQTT messages. If the rules take too much time, it will affect the performance and message throughput of the Broker itself. If there is a lot of data that needs to be persisted through the rule engine, it is recommended to increase the number of parallel=32 in /etc/nanomq.conf to increase the number of logical threads to support parallel processing of more rules and messages.

Edge Data Cache Configuration

Edge services often run in weak networks or harsh environments, and power outages often occur. In previous versions, NanoMQ's bridging feature supported caching of unacknowledged QoS 1/2 messages in local SQLite and automatic resending to avoid data loss. This release of v0.9.0 adds the handling of disconnected bridge connections, and adds many configuration options for edge scenarios to avoid running out of Flash life due to excessive cached data and overwriting Flash or writing too many times.

The new configuration options in /etc/nanomq_bridge.conf are:

 ## Enable sqlite cache
## Whether to enable sqlite cache
## 是否开启SQLite离线消息缓存功能
## Value: boolean
bridge.sqlite.enable=false

## Max message limitation for caching
## ( 0 means ineffective )
## Value: 1-infinity
## 最大缓存到磁盘/Flash的消息条数限制
bridge.sqlite.disk_cache_size=102400

## Mounted file path
## SQLite数据库文件路径
## Value: path
#bridge.sqlite.mounted_file_path=/tmp/

## The threshold of flushing messages to flash.
## flush msg to disk when reach this number
## Value: 1-infinity
## 数据刷盘的缓存窗口（消息条数），建议根据消息大小设置为 消息条数 * 消息大小 = page size
bridge.sqlite.flush_mem_threshold=100

## Resend interval (ms)
## The interval for resending the messages after failure recovered. (not related to trigger)
## 缓存消息的重发间隔
## Value: 1-infinity
bridge.sqlite.resend_interval=5000

At the same time, the 0.9.0 version no longer requires the user to modify the compilation parameters to enable SQLite, and now it is possible to control whether the Broker and bridging functions enable SQLite as a cache option through the configuration file. In addition, we have optimized the use of SQLite, enabled WAL mode, and adopted full synchronization to avoid file system damage. And starting from this version, the configuration options of the offline cache function of the bridge are separated from the default cache of the QoS message of the Broker. It is recommended that users who only need to automatically cache and resume data on the cloud bridge do not need to open SQLite in /etc/nanomq.conf Function.

NanoSDK: The first MQTT over QUIC SDK in C

On June 11, the IETF officially promulgated the HTTP/3 RFC technical standard document, and QUIC officially became one of the transport layer standards. IoT applications often encounter problems such as network roaming, frequent reconnection in weak network environments, and network congestion. Using QUIC's features such as streaming multiplexing, branch flow control, and lower connection establishment latency, these problems can be solved. significantly improved. NanoSDK 0.6.0 is the first to implement the first MQTT over QUIC SDK in C language based on the MsQuic project. It is recommended to use it with EMQX 5.0, the world's first Broker that supports MQTT over QUIC.

By adding QUIC support to the transport layer of NNG, NanoSDK enables protocols such as MQTT and nanomsg to be converted from TCP to UDP, thereby providing a better IoT connection experience. It internally maps and binds QUIC Stream and MQTT connection, and has built-in 0RTT fast handshake reconnection function.

In terms of API, the previous usage habits are maintained, and a single line of code can create an MQTT client based on QUIC:

 ## Create MQTT over Quic client with NanoSDK
nng_mqtt_quic_client_open(&socket, url);

Please refer to the message sample code:

https://github.com/nanomq/NanoSDK/tree/main/demo/quic , after compiling, you can use the following command to connect to port 14567 of EMQX 5.0 for testing.

 quic_client sub/pub mqtt-quic://54.75.171.11:14567 topic msg

Other function optimizations and bug fixes

In addition, NanoMQ 0.9.0 has the following bug fixes and update optimizations:

1. Added configuration options for the maximum received message length and the maximum forwardable message length to facilitate sending and receiving large messages.
2. Fixed the case where the nanomq broker restart restart command did not take effect.
3. Removed compilation requirement for C++ compiler.
4. Fixed an issue where receiving a client Disconnect message would not clear Session and Last Wish messages.
5. Fixed MQTT over WebSocket service not working properly in v0.8.0.
6. Fixed an issue where client unsub would cause the service to stop.
7. Fixed the data race problem caused by the sudden end disconnection of the Sub client when the client published a large number of messages causing a backlog.

coming soon

NanoMQ will officially release the 0.10 stable release version next month, and support the MQTT multi-channel bridge function. This feature is currently in the Demo stage and is available in the latest master branch. Users can compile and install by themselves, welcome to try: https://github.com/emqx/nanomq/ .

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/nanomq-newsletter-202206