Architecture Design of Million-level Message Throughput of Internet of Vehicles Platform

In this special series of articles, we will share with you how to build a reliable, efficient vehicle that meets the needs of industry scenarios based on EMQ's practical experience in the field of Internet of Vehicles, from theoretical knowledge such as protocol selection to practical operations such as platform architecture design. Internet platform.

foreword

In the previous article, we mentioned that the Internet of Vehicles TSP platform has many different business topics, and introduced how to design MQTT topics according to different business scenarios. Vehicles will continue to generate massive amounts of news, and every piece of data reported through the Internet of Vehicles is very precious, and there is huge business value behind it. Therefore, the vehicle TSP platform we build usually needs to have tens of millions of topics and a million-level message throughput capacity.

It is difficult for the traditional Internet system to support the throughput of millions of messages. In this article, we will mainly introduce how to design the architecture of the next-generation Internet of Vehicles platform for the requirement of millions of message throughput.

Related factors of message throughput design in IoV scenarios

The messages of the Internet of Vehicles are divided into uplink and downlink. Uplink messages are generally messages such as alarms issued by sensors and vehicles, and send device information to the cloud-based messaging platform. Downlink messages generally include remote control instruction set messages and message pushes, which are sent by the cloud platform to the vehicle.

In the design of IoV message throughput, we need to focus on the following factors:

message frequency

During the driving process of the car, GPS, on-board sensors, etc. are constantly collecting information. In order to receive real-time feedback information, the information it reports and receives is also very frequent. The reporting frequency generally ranges from 100ms to 30s, so when the number of vehicles reaches millions, the platform needs to support millions of messages per second.

message packet size

The message is to collect its own environment and status information through various sensors (new energy national standard data and enterprise standard data are common in the Internet of Vehicles scenarios). The size of the entire message package generally ranges from 500B to dozens of KB. When a large number of message packets are reported at the same time, the IoV platform needs to have a stronger ability to receive and send large message packets.

message delay

During the driving process of the vehicle, the message data can only be transmitted through the wireless network. In most IoV scenarios, the latency requirement for vehicles is at the ms level. The platform also needs to maintain low-latency message transmission under the condition of meeting the million-level throughput.

Topic number and level

When considering a million-level message throughput scenario, it is also necessary to standardize the design for the number of message topics and the topic tree level.

Payload codec

When the message packet is relatively large, it is necessary to focus on the encapsulation of the message body. Pure JSON encapsulation is not efficient enough for message parsing. You can consider using Avro, Protobuf and other encoding formats for Payload format encapsulation.

For a million-level message throughput scenario, the traditional architecture based on MQTT client sharing subscription messages or writing to a relational database in real time through a rule engine is obviously unsatisfactory. There are currently two mainstream architecture options: one is message access product/service + message queue (Kafka, Pulsar, RabbitMQ, RocketMQ, etc.), the other is message access product/service + time series database (InfluxDB, TDengine) , Lindorm, etc.) to achieve.

Next, we will introduce the implementation methods of these two architectures with the cloud native distributed IoT message server EMQX as the message access layer based on the above-mentioned correlation factors and the best practices of customer cases.

EMQX+Kafka builds a million-level throughput car networking platform

Architecture design

As one of the mainstream message queues, Kafka has the ability to store persistent data, perform persistent operations, and prevent data loss by persisting data to hard disk and replication. The back-end TSP platform or big data platform can subscribe to the desired messages in batches.

Since Kafka has the ability to subscribe and publish, it can not only receive from the south, and cache the reported messages; it can also transmit the instructions to be sent to the front end through the interface through the northbound connection for instruction delivery.

We take Kafka as an example to build an EMQX+Kafka million-level throughput IoV platform:

1. The connection and message of the front-end vehicle can be used for domain name forwarding through the load balancing product provided by the public cloud provider. If TLS/DTLS security authentication is adopted, four HAProxy/Nginx servers can be established on the cloud for certificate offloading and load balancing. .
2. Ten EMQXs are used to form a large cluster, and the one million message throughput is equally divided into one hundred thousand message throughput per node, while meeting the requirements of high availability scenarios.
3. If there are offline offline/message caching requirements, Redis can be used as the storage database.
4. Kafka acts as the overall message queue, and EMQX forwards all messages to the backend Kafka cluster through the rule engine.
5. The back-end TSP platform/OTA and other applications receive corresponding messages by subscribing to Kafka topics, and the control instructions and push messages of the business platform can be delivered to EMQX through Kafka/API.

Overall Architecture Diagram

In this solution architecture, EMQX as a message middleware has the following advantages, which can meet the needs of this scenario:

• It supports tens of millions of vehicle connections and millions of message throughput.
• The distributed cluster architecture is stable and reliable, and supports dynamic horizontal expansion.
• Powerful rule engine and data bridging and persistence capabilities, support millions of message throughput processing.
• With rich API and authentication systems, it can be smoothly connected.

Million throughput scenario verification

In order to verify the throughput capability of the above architecture, if conditions permit, we can build a million-level message throughput test scenario through the following configuration. The stress test tool can be selected from Benchmark Tools, JMeter or XMeter test platform. A total of 1 million devices were simulated, and each device had its own topic. Each device sent a message every second, and the pressure test continued for 12 hours.

The pressure test architecture diagram is as follows:

The results of the performance test part are presented:

EMQX Cluster Dashboard Statistics

In the EMQX rule engine, it can be seen that the processing speed of each node is 100,000/second, and the total speed of 10 nodes is 1 million/second.

EMQX rule engine statistics

1 million writes per second can be seen in Kafka, and storage continues all the time.

Kafka management interface statistics

EMQX+InfluxDB builds a million-level throughput car networking platform

Architecture design

Using the architecture of EMQX+ time series database, it is also possible to build a million-level message throughput platform. In this article, we take the InfluxDB time series database as an example.

InfluxDB is a high-performance time series database, which is widely used in scenarios such as monitoring data of storage systems and real-time data in the IoT industry. It records messages from the time dimension, has strong writing and storage performance, and is suitable for big data and data analysis. The analyzed data can be provided to the background application system for data support.

In this architecture, the EMQX rule engine is used for message forwarding, InfluxDB is used for message storage, and it is connected to the back-end big data and analysis platform, which can more conveniently serve time series analysis.

1. The messages of the front-end devices are used for domain name forwarding and load balancing through the cloud vendor's load balancing product.
2. This time, 1 EMQX is used as the test. If necessary, a multi-node method can be used to form a corresponding cluster solution (10 EMQX clusters can be deployed for the test of 1 million).
3. If there are offline offline/message caching requirements, Redis can be used as the storage database.
4. EMQX forwards all messages to the back-end InfluxDB through the rule engine for persistent data storage.
5. The back-end big data platform receives corresponding messages through InfluxDB, performs big data analysis on them, and then transmits the desired information to EMQX through API.

Overall Architecture Diagram

Scenario Verification

As shown in the test architecture diagram, the XMeter press simulates 100,000 MQTT clients to initiate connections to EMQX, the new connection rate is 10,000 per second, and the client heartbeat interval (Keep Alive) is 300 seconds. After all connections are successful, each client sends a message per second with a QoS of 1 and a payload of 200B. All messages are filtered by the HTTP InfluxDB rule engine bridge and persistently sent to the InfluxDB database.

The test results are presented as follows:

EMQX Dashboard Statistics

EMQX rule engine statistics

InfluxDB database receives data

EMQX Dashboard message statistics

A single EMQX server realizes the ability of a single server's message throughput of 100,000 TPS to persist to InfluxDB. Referring to the test scenario of the EMQX+Kafka architecture, expanding the EMQX cluster nodes to 10 can support 1 million TPS message throughput.

Epilogue

Through this article, we introduce the factors that need to be considered in the design of message throughput in the Internet of Vehicles scenario, and provide two more mainstream platform architecture designs for millions of throughput. Faced with the increasing amount of data in the Internet of Vehicles scenario, I hope this article can provide reference for relevant teams and developers in the design and development of the Internet of Vehicles platform.

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/million-level-message-throughput-architecture-design-for-internet-of-vehicles