Technology Inventory: The Past, Present and Future of Messaging Middleware

author: Lin Qingshan (flower name: Longji)

Operating system, database, and middleware are the troika of basic software, and message queue is one of the most classic middleware, with a history of more than 30 years. Its development has mainly gone through the following stages:

Phase 1, before 2000. In the 1980s, the first message queue was The Information Bus, which first proposed the publish-subscribe model to solve the communication problem between software; in the 1990s, it was the era of international business software giants, IBM, Oracle, Microsoft have Launched its own MQ, the most representative of which is IBM MQ, which is expensive and aimed at high-end enterprises, such as large financial, telecommunications and other enterprises; this type of commercial MQ generally adopts high-end hardware, software and hardware all-in-one delivery, and the architecture of MQ itself It is a stand-alone architecture.

II, 2000-2007. entered the 1900s, the first generation of open source message queues rose, and two major standards, JMS and AMQP, were born. The two corresponding implementations were ActiveMQ and RabbitMQ. Open source greatly promoted the popularity of message queues and lowered the threshold for use. , has gradually become the standard configuration of enterprise-level architecture. Compared with today, this type of MQ is mainly for traditional enterprise-level applications, for small traffic scenarios, and the horizontal expansion capability is relatively weak.

III, 2007-2018. explosive development of PC Internet and mobile Internet. Because the traditional message queue cannot withstand the access traffic of billions of users and massive data transmission, the Internet message middleware was born. The core capability is to fully adopt a distributed architecture and have strong horizontal expansion capabilities. The typical representatives of open source are Kafka, RocketMQ, There is also Taobao's Notify. The birth of Kafka also extends the message middleware from the Messaging field to the Streaming field, from the asynchronous decoupling scenario of distributed applications to the stream storage and stream computing scenarios in the big data field.

Phase 4, 2014~present. IoT, cloud computing, cloud native lead new technology trends. For IoT scenarios, message queues begin to communicate from server-side applications in the cloud to edge computer rooms and IoT terminal devices. Supporting IoT standard protocols such as MQTT has also become a standard configuration for major message queues.

With the popularity of cloud computing, the concept of cloud native is deeply rooted in the hearts of the people, and various cloud native representative technologies emerge in an endless stream, including containers, microservices, serverless, service mesh, event-driven, etc. The core issue of cloud native is how to redesign applications so as to fully release the technical dividends of cloud computing and achieve the shortest path to business success.

As one of the PaaS services of cloud computing, the message queue itself needs to further exert its "decoupling" capability to help businesses build modern applications. The most critical capability evolution here is the evolution of Eventing. By sublimating messages into "events", it provides standard CloudEvent-oriented orchestration and filtering, publish-subscribe and other capabilities to build a wider range of decoupling, including decoupling of cloud service events and business applications, decoupling of cross-organizational SaaS business events, legacy Decoupling of applications and modern applications, etc. At the same time, event-driven is also naturally in line with the paradigm of cloud computing serverless function computing, and is a catalyst for the evolution of application serverless.

For message middleware, cloud native has another meaning, which is the cloud-native evolution of the message queue's own architecture. How to make full use of the cloud's elastic computing, storage, and network to gain stronger technical indicators and serverless elastic capabilities .

What are the technological advances and breakthroughs in message middleware?

Alibaba Cloud MQ is a one-stop messaging service based on RocketMQ. With RocketMQ as the unified core, it implements industry-standard and mainstream messaging protocols, including MQTT, Kafka, RabbitMQ, AMQP, CloudEvent, HTTP, etc., to meet the demands of customers in diverse scenarios. In order to improve ease of use, we have productized different protocols and provided message services (such as Alibaba Cloud RabbitMQ, Alibaba Cloud Kafka) as independent products, out-of-the-box, O&M-free, and a complete observable system , to help open source customers seamlessly migrate to the cloud.

After continuous refinement of tens of thousands of enterprise customers in diversified scenarios and years of production practice of ultra-large-scale cloud computing, our core RocketMQ has gradually evolved to an integrated architecture and a cloud-native architecture.

1. Integrated Architecture

Technology trends such as microservices, big data, real-time computing, IoT, and event-driven continue to expand the business boundaries of messages. There are different message queues in the industry to meet different business scenarios. For example, RabbitMQ focuses on microservice scenarios, while Kafka focuses on It satisfies big data and event flow scenarios, while EMQ satisfies IoT vertical field scenarios. With the deepening of digital transformation, customers' business often involves cross-scenarios at the same time. For example, messages from IoT devices or business messages generated by microservice systems need to be calculated in real time. If multiple systems are introduced, additional Costs of machines, operation and maintenance, learning, etc.

In the past, "dividing" was often a compromise of technical realization, but now "combining" is the real demand of users. RocketMQ 5.0 expands diversified indexes based on unified Commitlog, including time index, million queue index, transaction index, KV index, batch index, logical queue and other technologies. At the same time, it supports RabbitMQ, Kafka, MQTT, edge lightweight computing and other product capabilities in the scene, and truly realizes the "message, event, flow" and "cloud edge" integrated architecture.

2. Cloud-native architecture

Cloud-native architecture refers to the architecture that is native to the cloud. Cloud computing is the "power source" of cloud-native. Talking about cloud-native without cloud computing is like talking on paper. In the past few years, RocketMQ has been based on Alibaba Cloud's ultra-large-scale cloud computing production practice, and has helped tens of thousands of enterprises to complete the digital transformation experience to absorb nutrients, thereby completing the evolution of Internet message middleware to cloud native message middleware. This is also the biggest difference between RocketMQ and other message middleware. It is a cloud-native architecture that has been practiced. Let's take a look at the key technological evolution of RocketMQ in the cloud-native architecture.

RocketMQ was born in Taobao's core e-commerce system in 2011. At the beginning, it was positioned to serve group business and designed for a single super-large Internet enterprise. The original architecture does not meet the cloud computing scenario well, and has many pain points, such as heavy-duty SDK, complex client logic, high development cost of multi-language SDK, slow iteration of business features; poor flexibility, computational storage coupling, customer The number of terminals and physical queues are coupled, and the number of queues cannot be expanded to millions or tens of millions; and other mainstream open source messaging projects have also not undergone cloud-native architecture transformation. Faced with a large number of data copy balancing, etc., it is not suitable for providing elastic services for large-scale customers in the public cloud.

To this end, RocketMQ 5.0 is redesigned for cloud computing scenarios, expecting to solve fundamental problems from the architectural level, and comprehensively upgrade clients, brokers and storage engines:

Client lightweight. RocketMQ 5.0 SDK sinks a lot of logic to the server, reduces the number of lines of code by two-thirds, and greatly reduces the cost of developing and maintaining multilingual SDKs; lightweight SDKs are easier to be integrated by cloud-native representative technologies such as Service Mesh and Dapr .

Separable memory-computation architecture. users can start the storage and computing functions in the same process or deploy them separately according to different scenarios. Separately deployed computing nodes can be "stateless", and one access point can proxy all traffic. Combining the new hardware kernel bypass technology on the cloud can reduce the performance and latency problems caused by separate deployment. However, choosing the "integrated storage and computing" architecture has the advantage of "nearest computing" and better performance. In the scenario of multi-tenancy, multi-VPC complex network, and multi-protocol access on the cloud, using the storage-computing separation mode can avoid direct exposure of back-end storage services to clients, and facilitate traffic control, isolation, scheduling, authority management, Protocol conversion, etc.

However, there are advantages and disadvantages. The separation of storage and computing also brings problems such as longer links, increased latency, and increased machine costs. The operation and maintenance have not been simplified. In addition to the operation and maintenance of stateful storage nodes, more operations are required. Dimensional stateless compute nodes. In fact, in most simple message sending and receiving scenarios, the data link basically consists of writing and reading without complex computing logic (computing logic is too simple compared to databases). High performance and low latency. Especially in big data transmission scenarios, the integration of storage and computing can greatly reduce machine and traffic costs, which can also be confirmed by the evolution of Kafka's architecture. In general, do not separate for the sake of separation of storage and computing, but return to the essential demands of customers and business scenarios.

Elastic Storage Engine. oriented to the scenarios of massive IoT devices and large-scale small customers on the cloud. We introduce the KV index of LSM to realize the capability of single-machine massive queues, and the number of queues can be expanded infinitely; in order to further release the capabilities of cloud storage, we implement hierarchical storage and message storage duration. From 3 days to monthly and annual levels, the storage space can be expanded indefinitely, while hot and cold data are separated, and the cost of cold data storage is reduced by 80%.

Serverless. In the old architecture, the client perceives the physical queue, and the physical queue is bound to a fixed storage node and has a strong state. The expansion and contraction of brokers, clients, and physical queues are coupled with each other, and the load balancing granularity is at the queue level, which is very unfriendly to the technological evolution of serverless. In order to achieve extreme elastic serverless, RocketMQ 5.0 further decouples logical resources and physical resources.

In the Messaging/disordered message scenario, the client specifies a topic to send and receive messages out of order. The new architecture shields the client from the queue concept and only exposes the logical resource topic. The granularity of load balancing is from the queue level to the message level, which realizes the statelessness of the client and the decoupling of elastic scaling between the client and the server.

In the streaming/sequential message scenario, the client needs to specify a queue (also called a partition) under the topic to send and receive messages in sequence. In the new architecture, physical queues are shielded from clients, and the concept of logical queues is introduced. A logical queue is distributed on different physical storage nodes through horizontal and vertical fragmentation. Horizontal sharding solves the high-availability problem. Multiple shards of the same logical queue can be written randomly at multiple points. Based on the principle of Happen before, the order is guaranteed. To solve the problem, through multi-level queue mapping, the second-level expansion of zero data migration is realized, and the elastic scaling of logical resources and physical resources is decoupled.

How do you view ecological players in the message field?

Guided by the trends of cloud native, IoT, and big data, messaging has become a rigid need for modern application architectures, and the usage scenarios are more extensive, which can be applied to asynchronous decoupling of microservices, event-driven, IoT device data upstream and downstream, and big data stream storage , lightweight flow computing and other scenarios. Strong customer demand and active market have attracted many manufacturers to join the competition.

From a good point of view, the full competition of manufacturers will further activate innovation, cultivate more users, and jointly expand the news market, and users seem to have more choices;

From a bad point of view, some message queues that lose competition in the future will enter a period of stagnation and offline, and users' applications will face major migration and stability risks. Therefore, users are advised to do their best to meet their own business needs. It is possible to choose standard interfaces and protocols to access, or directly use the industry's de facto standard message queue.

What is the future development trend of message middleware?

With the continuous development of IoT and 5G networks, the growth rate of data volume is 28%. It is expected that by 2025, the number of IoT devices will reach 40 billion, entering the era of the Internet of Everything. In the Internet of Things era, the amount of message storage and computing will increase explosively, and the message system will face huge cost pressures. In the future message system, we need to deep into the dividends of new hardware, such as persistent memory, and other technologies, and use the combination of software and hardware to optimize deeply, further reducing the cost of storing and computing messages.

Another very important trend in the IoT era is edge computing . Gartner predicts that by 2025, 75% of data will be processed outside the traditional data center or cloud environment. Adapt to edge computing environments. This also means that the integrated architecture of the message middleware must have a good plug-in design and can achieve polymorphic output according to the characteristics of the scene. For example, the form of public cloud can be deeply integrated with the infrastructure of public cloud, making full use of cloud disks and object storage to enhance storage capacity, and integrating services such as log services and application monitoring to improve observability; while the form of edge computing uses minimal resources. The cost outputs the ability of core storage and lightweight computing, which is simple enough.

In recent years, the rapid development of cloud computing has benefited from the digital transformation of a large number of enterprises around the world, which enhances the competitiveness of enterprises through online business, business dataization, and data intelligence. Data transformation is also accompanied by the transformation of business thinking. More and more enterprises adopt an "event-driven" model to build business logic and digital systems.

Gartner predicts that more than 60% of new digital business solutions will adopt an "event-driven" model in the future. From a business perspective, an "event-driven" model can help companies respond to customers in real time, seize more business opportunities, and create growth. From a technical point of view, the "event-driven" architecture can link heterogeneous systems across organizations and environments in a dynamic, flexible, and decoupled manner, and is naturally suitable for building a large-scale cross-organizational digital business ecosystem.

In response to this trend, Messaing has evolved to Eventing, and the product form of EventBridge (EventBroker) has emerged. In EventBridge, the concept of "event" becomes a first-class citizen, and event publishers and subscribers are not coupled to any specific message queue SDK and implementation. EventBroker builds a more generalized publish-subscribe model around the standard CloudEvent specification, which can link all heterogeneous event sources and event processing targets across organizations and environments.

At present, the digital business ecology built with "event-driven" has just started. In the future, EventBridge will realize more powerful capabilities around the abstraction level of events, such as full-link observability of events, event analysis and calculation, low-code development and other features to help enterprises Fully implement the "event-driven" architecture in the cloud era.

about the author:

​Lin Qingshan (flower name: LONGi), senior technical expert of Alibaba Cloud, head of Alibaba Cloud messaging product line. An expert in the field of international messaging, dedicated to the research and exploration of messaging, real-time computing, event-driven, etc., to promote the evolution of RocketMQ cloud-native architecture and hyper-converged architecture.​