Architecture design of MQTT messaging platform for tens of millions of vehicles

In this series of articles, we will share with you how to build a reliable, efficient vehicle that meets the needs of industry scenarios based on EMQ's practical experience in the field of IoV, from theoretical knowledge such as protocol selection to practical operations such as platform architecture design. Internet platform.

foreword

With the advancement of the new four modernizations (electrification, intelligence, networking and sharing) in the entire automotive travel field, various car manufacturers are gradually building a car networking system with intelligent driving and intelligent networking as the core. The new generation of Internet of Vehicles system puts forward higher requirements for the platform architecture of underlying message collection, transmission and processing.

In the previous article of this series, " MQTT Protocol in the Internet of Vehicles Scenario", we have mentioned that the MQTT protocol is currently the most suitable communication protocol for building a data platform in the Internet of Vehicles scenario. Based on this, in this article, we will continue to discuss the collection and delivery of MQTT messages in the Internet of Vehicles scenario, and how to build a MQTT messaging platform of tens of millions of Internet of Vehicles, in order to provide platform architecture design reference for enterprise users who are engaged in the Internet of Vehicles business.

The basis of the Internet of Vehicles: message collection and transmission

Evolution of Internet of Vehicles Transmission Protocols

As we all know, vehicle-to-everything (V2X) refers to the interaction between vehicle and cloud, vehicle and network, vehicle and vehicle, vehicle and road, vehicle and human, vehicle and sensor equipment, etc., to realize the communication between vehicle and public network. The dynamic mobile communication system is a heterogeneous communication network established to meet the elements of efficiency, safety and management in every link related to the vehicle. The communication protocol running in it has become the key and core of the construction of the Internet of Vehicles system.

In the course of the development of the Internet of Vehicles, there are mainly two mainstream communication technologies, which have played a role in promoting the overall development of the Internet of Vehicles:

DSRC (DeDICated Short Range Communication, dedicated short-range communication) : It was developed by ASTM in 1992 for the business scenario of ETC. After years of improvement and iteration, it evolved into the IEEE (802.1X) vehicle networking communication technology standard . For quite a long time, DSRC technology has been the mainstream car networking communication protocol used in the mainstream production and consumer markets of the international automobile.

C-V2X (Cellular Vehicle to Everything, cellular vehicle networking communication): C-V2X relies on the existing cellular base station, in addition to supporting the direct communication of PC5, RSU and vehicle can pass 4/5G channel (using Uu interface) It is connected to the V2X platform to realize vehicle-road collaborative communication. Compared with DSRC, C-V2X is technically better, it enhances the security and confidentiality of communication, supports high network capacity, and can support high bandwidth and large data volume requirements.

The competition between DSRC and C-V2X technologies is fierce, and both are expected to become mainstream vehicle networking communication standards. At present, my country has the most complete 5G communication network infrastructure, so it is more inclined to adopt C-V2X (LTE-V, 5G-V2X) communication technology. Next-generation car networking architecture for autonomous driving.

The significance of message platform construction for the Internet of Vehicles

With the rapid development of the Internet of Vehicles construction today, all OEMs have formed a consensus: the purpose of the Internet of Vehicles construction is not networking for networking, nor networking for in-vehicle entertainment, networking is for data. With the Internet of Vehicles, there is data. With data, supplemented by a complete data governance and application system, everything is available.

And the target data of this business is not limited to the relevant data on the car side. In the V2X framework, it is necessary to solve the interconnection between vehicles and vehicles (V2V), vehicles and roads (V2R), vehicles and the Internet (V2I), vehicles and cloud (V2C), vehicles and people (V2H), etc. Comprehensive data collection and analysis of roads, clouds, networks, and people. The 5G-based C-V2X protocol and communication method provide basic capability guarantee for the construction of the entire system.

From traditional OTA applications to many new intelligent application scenarios such as smart cockpit, high-precision map adaptation, centimeter-level positioning, long-distance connection between vehicle and mobile terminal, mobile terminal message collection, vehicle-road cloud map, vehicle-road collaboration and many other new intelligent application scenarios, the Internet of Vehicles business is very important to the message platform. The demand for and data processing system has expanded from the original vehicle cloud to the overall architecture construction of people-vehicle-road-network-cloud, which has put forward higher requirements for the construction of the entire message platform.

How to build a message communication and transmission system architecture with massive connections, high concurrent throughput, and low latency to ensure the ubiquity, convenience, high availability, reliability, security and high concurrency of the entire system has become a The key to the construction of a new generation of Internet of Vehicles systems in autonomous driving and vehicle-road collaboration scenarios.

Architecture design of tens of millions of car networking message platform

Next, we will take EMQ 's IoV message platform and overall data processing solution as an example to introduce how to build a 10 million-level IoV message platform.

business challenges

Secure access to vehicle, road test unit and mobile terminal

The car side needs to cover new car-connected services such as car-machine data reporting, POI distribution, file push, configuration distribution, push message, operation care, etc. The mass message topics generated require more secure and stable access and transmission to realize message subscription and publishing . The road end needs to realize the secure access of the roadside RSU, the collection and transmission of messages, and the transmission of map data.

Real-time and reliability of large concurrent message delivery

Application scenarios such as high-precision maps, centimeter-level positioning, and vehicle-road coordination all need to solve the millisecond-level low-latency and high-reliability transmission capability guarantee of massive vehicle-road map messages. The message processing platform requires high performance, low latency, and high reliability. The ability to support tens of millions of connections and millions of concurrent business scenarios.

Rich application scenario

In the Internet of Vehicles system with autonomous driving as the core, it is necessary to use the message platform to connect various applications based on people, roads, maps, and clouds. Connect the vehicle-end data with application scenarios such as high-precision maps, centimeter-level positioning, vehicle-road coordination, and mobile phone connection through the message platform, ensure the consumption supply of applications through the message platform, and provide high-performance, low-latency and high-reliability. data schema.

Mass data storage, processing and

After the massive IoT data from people, vehicles, roads, clouds, graphs, and networks is collected, it is necessary to conduct full life cycle management for the access, storage, processing, and distribution of these large-scale real-time data streams to provide applications. Database support for dynamic and continuous data flow supports applications to deeply use car networking data to serve consumers and make business decisions.

Overall solution

In the solution, we mainly use EMQ's cloud-native distributed IoT access platform - EMQX to realize the data connection, movement and processing of the vehicle end, people and road ends in the Internet of Vehicles system. EMQX's integrated distributed MQTT message service and powerful IoT rule engine can provide a basic capability base for highly reliable, high-performance IoT real-time data movement, processing and integration, helping enterprises to quickly build critical business IoT platforms and applications.

Message processing for the car side

EMQX uses the MQTT protocol to access the connected car system. The vehicle terminal is connected to the EMQX distributed cluster through load balancing. The horizontal expansion capability of EMQX can realize the data communication capability of tens of millions of vehicle connections and millions of concurrent responses. Through the rule engine, one-stop capabilities such as massive message bridging message queues, persistent storage, offline message storage, etc. can be realized, while providing rich API atomic capabilities northbound integration.

In terms of security, EMQX not only supports TLS/DTLS or national secret GMSSL security protocol to ensure the reliability and stability of the system; it also provides multiple guarantee mechanisms such as heartbeat monitoring, testament message, QoS level, etc., through offline message storage to achieve real-time in a complex network environment , Safe and reliable vehicle-machine message communication.

Message processing for people and roadside

EMQX provides a message collection and processing platform for mobile APP, RSU and other terminals for people and road ends. Based on the 5G network slicing capability, through the nearby access of personal terminals and roadside units, ultra-low latency traffic information services are realized. Through MQTT and other protocols, the road condition information perceived by the human terminal and roadside facilities is pushed to the cloud control platform, and the cloud control platform integrates V2X algorithm to realize intelligent traffic scenarios such as road collaborative perception, safety reminder, and remote collaborative control.

In terms of security, it supports the international standard TLS/DTLS encryption or the national encryption algorithm GMSSL encryption, and guarantees the secure communication of the coordination of the human-vehicle-road information system by extending the PKI/CA certificate authentication system.

10 million message access framework model

For the new generation of Internet of Vehicles scenarios, the overall message access and data processing platform reference architecture of EMQ with tens of millions of connections and millions of concurrency is as follows:

• Business Scenario : Vehicles, mobile APPs, roadside RSUs and other devices in the Internet of Vehicles system are connected through MQTT, enabling concurrent access to more than ten million terminals.
• system architecture: terminal devices access through MQTT, HTTP and other protocols, and connect to the distributed messaging platform EMQ X through load balancing components. Through distributed multi-cluster deployment to meet the requirements of tens of millions of concurrent connections, according to the throughput capacity of millions of messages, the rules engine is connected to the Kafka cluster to realize data forwarding. The Internet of Vehicles service platform, high-precision map service, V2X cloud control service, positioning service and other vehicle network-related applications can be directly consumed by subscribing to Kafka data. Meanwhile, EMQ provides three southbound interface services: REST, MQTT and MQ message queue. Two-way communication of vehicle control (remote control) messages.

Through the above reference framework, EMQ can achieve the business requirements of tens of millions of connections and millions of concurrent throughputs in the Internet of Vehicles scenario through the EMQX cloud-native distributed IoT access platform.

Tens of millions of messages access test

Test environment and purpose

A car company plans to verify the following capabilities of the EMQX cluster based on the test environment in the Internet of Vehicles scenario, and prepare the corresponding technical architecture and capability support for subsequent business growth:

• It can support 10 million concurrent connections, and support QoS 0 messages of 100,000 to 150,000 per second, with a payload of 100 bytes, bridged to Kafka through the rule engine;
• 10 million concurrent connections to subscribe and consume OTA broadcast topics;
• 3 million users connecting at the same time will not cause cluster avalanche and test the time it takes to connect.

In addition, after completing all the above tests, continue to explore the highest throughput of message sending and bridging forwarding to Kafka that can be supported under the current configuration with 10 million concurrency (increase the client message sending frequency according to the EMQX cluster resource usage), and Tested to meet the highest message throughput with a QoS of 2 and an average response time of 50 ms under 10 million connections.

test preparation

The client connects to the load balancing ELB through TLS encryption, then TLS terminates the client at HAProxy, and finally connects to the EMQX cluster through TCP. By terminating TLS on HAProxy, the support capability of the EMQX cluster can be improved. In this deployment mode, the processing capability of EMQX is exactly the same as the client's direct connection through MQTT TCP. On the other hand, compared with the MQTT TCP connection, the client needs to consume more resources through the TLS connection. The scale of this test is tens of millions, and the number of test machines required is large. In order to reduce the required test resources, it does not Affects the test target of EMQX cluster, this test will use TCP connection directly.

Service	quantity	version	OS	specification	CPU	RAM	network card	port
Load Balancing Cloud Service	1			Network/Large II				18083/1883/8081
EMQX node	10	V4.3.4	Centos7.8	c6.16xlarge.2 ordinary block storage	64C	128G	1	18083/1883/8081/8883
Kafka cloud service	4	2.3.0	Centos7.8	c6.4xlarge.2 ultra-high IO block storage	16	32G	1	9092
XMeter Stress Test Control Node	2	3.0	Centos7.8	c6.4xlarge.2	16	32G	1	443/80/3000/8086 Intranet release
XMeter Stress Test Node	43	3.0	Centos7.8	c6.4xlarge.2	16	32G	5	Intranet release

testing scenarios

serial number	scene name	describe	Desired result
1	Ten million connections + message throughput	10 million MQTT TCP concurrent connections with a heartbeat interval of 200s. Among them, 7 million are background connections (only connect without sending messages), 3 million active users, each user reports a QOS 0 message every 15S, and the payload is 100B. Messages are bridged to Kafka via the rules engine. 1 hour test first, 24 hours stability test after passing	The success rate of the intranet test is 100%, there is no message backlog, the CPU and memory are stable during the test, and there is no large jitter.
2	message broadcast	10 million MQTT TCP concurrent connections, all connections subscribe to the same OTA broadcast topic (QoS 0, payload is 100B). Simulate an MQTT client to broadcast a message to the topic every 10 minutes, test for 30 minutes	The success rate of the intranet test is 100%, and all subscribing clients successfully consume 3 messages
3	3 million concurrent transient connections	3 million MQTT clients initiate connections at the same time, and test the time required for all connections to complete	3 million clients are successfully connected, and the cluster will not avalanche
4	Exploration of the highest message throughput under 10 million connections	The highest message throughput rate (Qos 0, payload 100B/1kB) that can be achieved under the existing configuration and 10 million connections and bridging kafka	After the highest message throughput rate is reached, the test is performed for 2 hours. The success rate of the intranet test is 100%. There is no message backlog. The CPU and memory are stable during the test, and there is no large jitter.
5	TPS at Average Response Time	Under 10 million connections, the message is QoS2, the payload is 100B, and the average response time is within 50 mm. The highest message TPS supported	Able to achieve a throughput capacity of not less than 200,000 TPS

Test Results

The following are the results of this test:

serial number	scene	Average Response Time	EMQX node CPU usage	EMQX Node CPU IDLE	EMQX node memory usage (G)	Bandwidth required for LB (MB)
1	10 million connections + 200,000 message throughput, QoS 0, payload 100B	1.5ms	31%~48% Average 47%	37%~54% Average 47%	Used: 27.7~42 Free: 78.2~92.5	45
2	Message broadcast under 10 million connections	100ms	up to 21%	Minimum 69%	Used Max 32.3 Free Min 87.9	200
3	3 million client instant connections	3 minutes to complete the connection	up to 25%	Minimum 63%	Used Max 14.7 Free Min 108.2	400
4	Explore the highest throughput: 10 million connections + 1.2 million message throughput, QoS 0, payload 1kB	164.3ms	23%~64% Average 46%	20%~64% Average 43%	Used: 33~38 Free: 81.3~87.1	1350
5	10 million connections + QoS2 200,000 message throughput, payload 100B	51.4ms	3%~51% Average 41%	31%~53% Average 43%	Used: 22.2~29 Free:91~98	95

summary

As shown in the above results, under the current deployment architecture, it can meet the verification requirements of the car company for 10 million concurrent connections + 200,000 messages bridged to Kafka, message broadcasting, and 3 million instantaneous concurrent connections. In the exploratory test, the maximum TPS of 1.2 million messages (QoS 0, payload 1kB) was tested under 10 million connections. The test lasted for 10 hours. The EMQX cluster was stable, the CPU idle was as low as 20%, and the memory usage was stable.

It can be seen from the above that EMQX has outstanding performance in supporting tens of millions of connections in the Internet of Vehicles scenario, and the architecture is stable and reliable.

Introduction and use of stress testing

Due to the large number of test machines required and the complicated management, this test was conducted using the XMeter performance test platform and the JMeter-MQTT plug-in of the commercial version of EMQ's test software.

XMeter is an extended performance testing platform based on the open source testing tool JMeter. In view of the characteristics of the Internet of Things, such as large access scale, elastic expansion requirements, multiple access protocols, and mixed scenarios, XMeter has transformed JMeter to support large-scale, high-concurrency performance testing, such as the realization of 10 million-level MQTT Concurrent connections and message throughput testing. In addition to testing the MQTT protocol, it can also support the testing of mainstream applications such as HTTP/HTTPS.

JMeter-MQTT plug-in is an open source MQTT performance test plug-in implemented by XMeter. It has been used in many projects and is currently the most popular MQTT plug-in in the JMeter community.

write at the end

Through this article, we introduce the architecture design of the MQTT messaging platform for the Internet of Vehicles with tens of millions of levels based on the cloud-native distributed IoT access platform EMQX, and verify the performance of the architecture in the environment of tens of millions of concurrent connection scenarios. The construction of the system's message data platform provides a possible design reference.

As the world's leading provider of IoT data infrastructure software, EMQ is committed to building high-performance, low-latency, high-availability, and high-reliability products, and providing overall solutions for message collection, movement, processing, and analysis for the next-generation Internet of Vehicles system The solution provides infrastructure service guarantee for the autonomous driving and intelligent networked vehicle business of vehicle manufacturers, T1 suppliers, after-market service providers, travel service companies and government management agencies, and realizes the intelligent connection of people, vehicles, roads and clouds .

About the author

Fan Rongjie, Director of EMQ Solution Architecture. With rich experience in the field of IT infrastructure, he has been deeply involved in enterprise digital transformation and IoT architecture construction, data platform construction, agile development, lean operation and maintenance, and DevOps concepts for many years. service to market.

Other articles in this series

• MQTT protocol in car networking scenarios

Copyright statement: This article is original by EMQ, please indicate the source when reprinting.

Original link: https://www.emqx.com/zh/blog/mqtt-messaging-platform-for-internet-of-vehicles