MQTT over QUIC: The Next-Generation IoT Standard Protocol Brings New Impetus to Messaging Scenarios

Introduction: The first groundbreaking product to bring QUIC to MQTT

Not long ago, the open source large-scale distributed IoT MQTT message server EMQX released version 5.0 . EMQX 5.0 is not only the world's first distributed MQTT message server that supports 100 million connections in a single cluster, but also pioneered the introduction of QUIC support.

QUIC is the underlying transport protocol of the next-generation Internet protocol HTTP/3. Compared with the TCP/TLS protocol, it provides an efficient and flexible transport layer for the modern mobile Internet while reducing connection overhead and message delay.

Based on the advantages of QUIC, which are extremely suitable for IoT message transmission scenarios, EMQX 5.0 introduces QUIC support (MQTT over QUIC) and designs a unique message transmission mechanism and management method.

This article will show you the advantages and value of this leading technology for IoT scenarios through a detailed analysis of MQTT over QUIC, and help you more effectively use EMQX 5.0's support for QUIC in various MQTT application scenarios. More efficient and low-cost IoT data transmission.

What is QUIC

QUIC is a general-purpose transport layer network protocol built on top of UDP, originally proposed by Google as an alternative to TCP+TLS to improve user experience.

QUIC has many advantages over existing TLS over TCP schemes:

• Quickly establish low-latency connections (1 RTT or 0 RTT)
• End-to-end encryption, handshake authenticated via TLS 1.3
• Multiplexing to avoid head-of-line blocking
• Improved congestion control, pluggable congestion control strategies
• Multi-path support, smooth connection migration
• Stateless Load Balancing
• Existing network can be supported without modification and upgrade

Because of its efficient transmission efficiency and multi-channel concurrency capability, QUIC has become the underlying transmission protocol of the next-generation Internet protocol HTTP/3.

Introduction to HTTP/3 Protocol

In October 2018, the IETF's HTTP and QUIC working groups jointly decided to call the HTTP mapping over QUIC HTTP/3 in advance of making it a global standard. On June 6, 2022, the IETF standardized HTTP/3 as RFC ) 9114 .

The goal of HTTP/3 is to provide fast, reliable, and secure Web connections on all forms of devices by addressing the transport-related issues of HTTP/2. HTTP/3 uses similar semantics to the HTTP/2 version, including the same request methods, status codes, and message fields. The fundamental difference between the two is that HTTP/2 uses the TCP/TLS protocol at the bottom, while HTTP/3 uses the QUIC protocol.

According to W3Techs statistics , at least 40% of the Internet traffic is based on QUIC, and 25% of the top 10 million websites already support the HTTP/3 protocol, including top streaming sites such as Google, Youtube, and Facebook.

The application prospect of QUIC in MQTT communication scenarios

MQTT is a communication protocol for the Internet of Things based on TCP. The compact packet structure can achieve stable transmission on severely limited hardware devices and low-bandwidth and high-latency networks; many features such as heartbeat mechanism, testament message, and QoS quality level can cope with Various IoT scenarios.

However, due to the limitations of the underlying TCP transport protocol, the MQTT protocol has inherent drawbacks in some complex network environments:

• Network switching causes frequent connection outages
• Difficulty in re-establishing a connection after a network disconnection: The operating system releases resources slowly after a disconnection, and the application layer cannot sense the disconnection state in time, and the server/client overhead is high when reconnecting
• Data transmission in weak network environment is limited by congestion, packet loss detection and retransmission mechanisms

For example, Internet of Vehicles users usually face similar problems: vehicles may run in mountains, mines, tunnels, etc., when they enter the dead zone of the signal or passively switch base stations, the connection will be interrupted, frequent connection interruptions and slower connection recovery. Speed can lead to poor user experience. In some businesses that require high real-time and stable data transmission, such as L4 level unmanned driving, customers need to spend a lot of costs to alleviate this problem.

In scenarios like the above, QUIC's low connection overhead and multi-path support show its advantages. After more in-depth exploration, we believe that MQTT Over QUIC can solve this dilemma very well. Based on the QUIC 0 RTT/1 RTT reconnection/new creation capability, it can effectively improve the user experience in weak networks and unstable network paths.

MQTT over QUIC implementation of EMQX 5.0

The current implementation of EMQX replaces the transport layer with QUIC Stream, the client initiates the connection and creates the Stream, and EMQX and the client interact on a bidirectional Stream.

Considering the complex network environment, if the client fails to pass the QUIC handshake for some reason, it is recommended that the client automatically fall back to traditional TCP to prevent the system from being unable to establish communication with the server.

At present, the following features have been implemented in EMQX 5.0:

• More advanced congestion control: effectively reduce the data packet loss rate, and can still transmit data continuously and stably in the case of network fluctuations during the test
• Operation and maintenance friendly: reduce the overhead caused by large-scale reconnection (time overhead, client/server performance overhead), and reduce system overload caused by unnecessary application layer state migration (0 RTT)
• More flexible architecture innovation: For example, Direct server return (DSR, direct server return mode), only ingress/request traffic goes through LB, egress and response traffic bypass LB and return directly to the client, reducing LB bottlenecks
• Reduced handshake latency (1 RTT)
• Multi-path support, smooth connection migration: switch from 4G to WIFI, or change the quintuple due to NAT Rebinding, QUIC can still continue the connection state on the new quintuple, especially for mobile devices with frequent network changes
• More agile development and deployment: The protocol stack is implemented in userspace, enabling rapid development and iteration
• End-to-end encryption: Unencrypted packet headers carry very little information, reducing the impact of intermediate nodes in the transmission path, bringing better security and a more controllable user experience

At the same time, there are more capabilities to be further explored:

• Streams of different topics: For independent topics, each topic can have independent Streams to eliminate the impact of long blocking on other topics, such as long blocking on the receiving end or flow control, and can also implement the priority topic function.
• Flows with different QoS: For example, in "flow control", QoS 0 transmissions should give way to high QoS transmissions.
• Divide control messages into different streams: MQTT control messages can be sent unidirectionally or bidirectionally. For example, the client can send an UNSUBSCRIBE request asynchronously through "control flow" to ask the server to stop sending data that is no longer of interest.
• More fine-grained coordinating flow control between transceivers: Flow control is performed for each flow and for the entire connection to achieve more fine-grained flow control.

QUIC vs TCP/TLS test comparison

In the laboratory environment, we simulated the performance of QUIC and TCP/TLS in different scenarios based on EMQX 5.0.

test environment

• Test platform: EMQX 5.0 single node
• Server Specifications: AWS EC2 M4.2xlarge (8 cores 32GB)
• OS: Ubuntu 20.04
• Number of clients: 5000
• loadgen parallel count: 8
• latency value: P95

Client connection delay

Test the delay of handshake, connection establishment, and subscription completion under different network delays. Compared with TLS, QUIC has certain advantages when the network latency is high.

0 RTT reconnection delay

After testing the disconnection, reconnect and resume the delay required for reconnection.

Since QUIC can carry the application layer data packet on the first packet in the 0 RTT scenario, the application layer responds faster than TCP with a back-and-forth handshake.

The QUIC protocol supports 0 RTT handshake. After the client and server complete the initial handshake, the server can send an NST packet to the client. Clients can use NST to quickly reconnect after a disconnection by skipping many of the steps in 1 RTT.

The advantage of 0 RTT is that it can effectively reduce the handshake overhead between the client and the server and improve the performance (handshake delay). EMQX sends NST packets to the client by default, and the validity period is 2 hours.

0 RTT also supports early data. Compared with 1 RTT, which requires handshake to complete the application layer transmission, the early data of 0 RTT can carry application layer data on the first packet to quickly restore or restart application layer services. . However, since early data at 0 RTT cannot prevent replay attacks, QUIC recommends not to carry data that will change the application state at 0 RTT.

EMQX does not support early data by default, this test is only used for comparison verification.

Test results show that QUIC outperforms pure TCP after the first handshake if the MQTT layer protocol is designed properly.

Server resource usage when connecting/reconnecting

Test the CPU and memory usage of the server in different processes of new connection and disconnected reconnection to compare the resource overhead of TLS, QUIC 1 RTT and 0 RTT handshake. Test results show that QUIC's CPU and memory usage are better than TLS, but rebuilding connections consumes more bandwidth than TLS.

Note 1: Mainly for MQTT clearing session, kicking off the extra overhead of old connections

Note 2: Mainly for a large number of QUIC initialization handshake packets caused by the MTU verification of the transmission path

Client address migration

This test simulates changes in business layer messaging during large-scale client address migrations.

Traditional TCP/TLS clients must sense disconnection at the application layer before reconnecting, which is very slow and involves many unnecessary retransmissions. QUIC's processing is smoother, and the transport layer maintains the connection without requiring reconnection and makes the application layer insensitive (the application layer can also subscribe to address changes if necessary).

QUIC has no effect on message sending when the client's source IP address/port changes. However, after the TLS connection is changed, the message sending is interrupted. Even if the client can reconnect to EMQX through the reconnection mechanism, the intermediate time window will not be able to perform any operations.

This result shows that QUIC is very suitable for use in environments where the network frequently needs to switch.

Network Packet Loss Test

Test data transmission under weak network conditions. We did 3 tests respectively: EMQX terminated TCP/TLS, QUIC and nginx terminated TCP/TLS.

Test scenario: EMQX publishes QoS 1 messages at a rate of 20K/s, during which network errors are injected: 20% out of order (the order of the sender and receiver packets is inconsistent), 10% packet loss, additionally added in the QUIC test Network handoff interference every 30 seconds.

In this case, the data received by the QUIC server is slightly jittery, but the message is not lost; while TLS suffers from congestion and packet loss caused by poor network environment. This result shows that QUIC can provide reliable transmission in a weak network environment.

In the yellow circle mark, we have removed the network error, and we can see that the TLS sending and receiving has resumed normal sending and receiving, and the number of packets has not accumulated, while QUIC has only changed from a slight jitter to a smoother.

Easier to use: MQTT over QUIC SDK

NanoSDK 0.6.0 is the first to implement the first C language MQTT over QUIC SDK based on the MsQuic project.

By adding QUIC support to the transport layer of NNG, NanoSDK enables protocols such as MQTT and nanomsg to be converted from TCP to UDP, thereby providing a better IoT connection experience. It internally maps and binds QUIC Stream and MQTT connection, and has built-in 0 RTT fast handshake reconnection function.

Please refer to NanoSDK QUIC Demo for message sample code.

We will also package based on NanoSDK and launch SDKs in Python, Go and other languages in the near future, so that more users can experience the advantages of MQTT over QUIC as soon as possible.

At the same time, the relevant SDK will support QUIC fallback. When QUIC is unavailable, the connection layer will automatically switch to TCP/TLS 1.2 to ensure normal operation of services in various network environments.

Future EMQX QUIC

Combining QUIC features and IoT scenarios, we have planned many features for MQTT over QUIC, such as topic priority setting by distinguishing control channels, unreliable real-time streaming to cope with high-frequency data transmission scenarios, and flexible topics and data channels (Stream) mapping to reduce interference between topics. It will appear in future versions.

EMQ is also actively promoting the standardization of MQTT over QUIC. After becoming the only Chinese company with voting rights so far in the OASIS MQTT technical committee in 2018 and participating in the formulation of the 5.0 protocol standard, EMQ has also submitted a draft of MQTT over QUIC. It is believed that in the near future, the underlying protocol of MQTT will support both TCP and QUIC, benefiting the entire IoT industry.

Epilogue

It can be seen that QUIC is very suitable for a weak network environment where the UDP MTU size of a traditional TCP/IP network can be guaranteed or an environment where the network is frequently switched. QUIC has great potential for IoT scenarios where devices are always on the move (such as Internet of Vehicles, mobile acquisition, etc.), or scenarios that require frequent disconnection and are not suitable for long-term connections (such as devices that need to sleep regularly). , is a more suitable underlying protocol choice, which is why EMQX 5.0 introduced QUIC support.

The first implementation of MQTT over QUIC in EMQX 5.0 makes EMQ once again at the forefront of the global IoT message server field. EMQ will always adhere to the continuous iterative upgrade of products driven by continuous technological innovation, and look forward to bringing infrastructure guarantee and business innovation power to the Internet of Things field through leading products.

Other articles in this series

• EMQX 5.0 Product Interpretation 01 | How EMQX 5.0 under the New Architecture of Mria + RLOG Realizes 100 Million MQTT Connections

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/mqtt-over-quic