Analysis of the basic concepts of "message-driven, event-driven, and stream"

Author: Ken Meng

Alibaba Cloud's message queue RocketMQ 5.0 has achieved a new upgrade, realizing the integration from "message" to "message, event, and stream". Based on this, the three words Message-Driven, Event-Driven, and Streaming are the most recent news in the field. Frequent words, but because the concept is too new, many students do not really understand the similarities and differences here. This article rearranges the three concepts, and sorts out a relatively clear concept for everyone.

background

First, the three concepts are specifically translated as follows:

• Message-Driven: message-driven communication;
• Event-Driven: event-driven communication;
• Streaming: Streaming mode.

These three modes are similar to asynchronous communication modes. The service that sends the message will not wait for the consuming message service to respond to any data. Service decoupling is a common feature of the three modes;

As long as it is in the field of service communication, the following characteristics should also be considered when selecting models:

• Ordering: Is it possible to guarantee a specific order of delivery;
• Transactions: Whether producers or consumers can participate in distributed transactions;
• Persistence: how the data is persisted and whether the data can be replayed;
• Subscription filtering: Whether it has the ability to do subscription filtering based on Tag or other fields;
• At – least-once (delivered at least once), At-most-once (delivered at most once), Exactly-once (delivered exactly).

After the general background is introduced, let's take a look at what each model represents.

Message-Driven

In message-driven communication, the general link is that a message producer (Producer) sends a message to a message consumer (Consumer). The model is as follows:

Middleware is usually used in the message-driven mode. The more common middleware are RocketMQ, Kafka, RabbitMQ, etc. The purpose of these middleware is to decouple the two systems by buffering messages delivered by producers until consumers are ready to receive them.

In a message-driven architecture, the format of messages is formulated based on the needs of consumers; message delivery can be one-to-one, many-to-many, one-to-many, or many-to-one.

A common example of message-driven communication is the push of commodity orders. The upstream component is responsible for generating the order, and the downstream component is responsible for receiving and processing the order. In fact, the upstream generation of components through such a communication method does not need to care about the life cycle of the entire order, and focuses more on how to quickly generate orders, so that the performance of a single component can be improved. !

The message-driven mode provides light coupling between services (this part of the coupling refers to the Producer/Consumer SDK), and can extend production and consumption services according to requirements.

Event-Driven

First of all, a point of view must be stated: event-driven is actually an improvement of the message-driven method. It imposes strict restrictions on the size of the message body and the message format. This layer of message-based restriction encapsulation is actually called an event.

In the event-driven model, the producer publishes events to represent system changes, and any interested and authorized service can subscribe to these events and use these events as triggers to start some logic/storage/task.​

Event-driven patterns can be one-to-one, many-to-one, one-to-many, or many-to-many. Usually, multiple targets execute different events according to the filter conditions.

In an event-driven architecture, the event format is formulated by the producer according to the event standard protocol; due to more standardized restrictions and encapsulation, the event producer does not need to care about which systems are consuming the events it generates.

Events are not commands, events do not tell consumers how to process information, their role is only to tell consumers that an event has occurred at this moment; an event is an immutable data, important data, it is the same as the data value of the message ; Usually when an event occurs and executes, it is often accompanied by another event.

Event-driven provides minimal coupling between services and allows production and consumption services to be extended according to needs; event-driven can add various new components without affecting existing services.

Event-driven can also be a very apt example. We take "customer purchases a product" as an event to demonstrate the application in the event scenario:

• The CRM (customer relationship system) system receives the customer purchase information and can update the customer's purchase record by itself;
• EMR (Inventory Management System) The system receives customer purchase information, dynamically adjusts inventory and replenishes in time;
• The courier service receives the customer's purchase information, makes an order by itself, and informs the courier company for delivery.

Looking at it this way, can the event-driven pattern be applied and appear anywhere!

In the direction of EventBridge productization, it is precisely because of some standardized encapsulation for messages that it is possible to realize capabilities such as filter (filtering), transform (transformation), schema (event structure), and search (query) for the event itself. These capabilities also expand more event-driven scene functions and related features.

Streaming

A stream is an ordered set of unbounded events or data that is usually executed on a fixed segment of events (eg 00:00 – 12:00) or a relative event (eg last 12 hours).

Usually a single event is often the use of the event itself, but the possible operations for the stream are likely to be filtering, combining, splitting, mapping, and so on.

Stream operations can be stateless or stateful:

• Stateless for single event operations, including filtering and mapping;
• It is stateful depending on the time or position (eg offset, time) of the message in the stream. In stateful operations, the stream processing logic must reserve some memory for consumed messages. Stateful includes doing Batch Size, Batch Window, etc. to the data.

Flow can also give a relatively simple example. For example, our logistics system will generate an event when an item passes through a logistics node, but to find the complete flow status event of this item, it must be the aggregation of individual events of each logistics node. , then this aggregate event is a stream event.

Kafka is the most typical streaming middleware. In streaming scenarios, the location information of events is crucial. Usually the location information (Eg offset) is hosted by the consumer.

event specification standard

After talking about what Event and Streaming are, let's add a bit about their specifications.

The purpose of the event specification is to clarify the relationship between event producers and consumers. At present, there are two main parts: AsyncAPI and CloudEvents;

AsyncAPI: Provides the corresponding Open API and Swagger based on the event API; CloudEvents: Focuses on the metadata of processing events.

The following also focuses on some related conceptual references about CloudEvents: The core of CloudEvents is to define a set of metadata about events transmitted between different components, and how these metadata should appear in the message body.

Its gist is roughly as follows:

• event normalization;
• Reduce the difficulty of platform integration;
• Improve the portability of FaaS;
• source events are traceable;
• Improve event correlation

Only with accurate event bodies and event information can a more stable system architecture be made, and the awe of events will always be maintained.

With some terms and definitions:

Occurrence: Occurrence , refers to the logical occurrence of an event, based on a certain situation, the event occurs;

Event: Event, a data record representing the event and context. Routing can be determined based on the information in the event, but the event itself does not contain routing information;

Producer: The producer, the instance or component that actually creates the event;

Source: source, the context in which the event occurs, which can be composed of multiple producers;

Consumer: Consumer, receives events and consumes events;

Intermediary : an intermediary that receives a message containing an event and forwards it to the next receiver, similar to a router;

Context: context, context metadata is encapsulated into context attributes to determine the relationship between events and other systems;

Data: data, also called payload;

EventFormat : event format, such as json;

Message: message, encapsulates events and passes them from source to destination;

Protocol: Protocol, which can be an industry standard such as http, an open source protocol such as Kafka or a vendor protocol such as AWS Kinesis;

Protocol Binding: Protocol binding, describing how to send and receive events through a given protocol, and how to put events into messages.

Heavy recommendation

This article aims to help you have a clearer understanding and cognition of the high-frequency words "Message-Driven, Event-Driven and Streaming" in the recent message field. Among them, Event-Driven EDA as Gartner One of the top ten predicted technology trends, EventBridge, as the next generation of messaging middleware, is also one of the current key directions. If you are interested and want to know more, you can pay attention to the "Alibaba Cloud EventBridge Series Open Courses" , the complete course is in full swing!

Click here to watch the video of the course~