Alibaba Cloud EventBridge event-driven architecture practice

About We believe that EventBridge is a new computing driving force in the cloud-native era. These data can drive cloud computing capabilities and create more business value.

Author: Zhou Xinyu

review & proofreading:

Editing & Typesetting: Wen Yan

content of this article is compiled from the speech at the China Open Source Annual Conference

First, I will introduce myself. I am Zhou Xinyu, a PMC member of RocketMQ. I am currently responsible for the product development of Alibaba Cloud RocketMQ and EventBridge. Today my sharing mainly includes the following parts:

• Messages and events, microservices and event-driven architecture

• Alibaba Cloud EventBridge: Event-driven architecture practice

• Build Alibaba Cloud unified event hub based on RocketMQ kernel

• A new trend in the cloud-native era: Serverless+ event-driven

• The future of event-driven architecture

Messages and events, microservices and event-driven architecture

First of all, let's talk about the difference between messages and events: everyone knows the messages in RocketMQ. It is a very general concept and a more abstract concept than events. Because the content body of the message is a Byte array, without any definition, it is a weak Data, so it is a very general abstraction.

In contrast, events may be more concrete. In general, it has a Schema to accurately describe the fields of the event. For example, CloudEvents has a clear Schema definition for the event. Events also often represent the occurrence of something or the change of a certain state, so they are very concrete.

In terms of usage, messages are often used in the asynchronous decoupling architecture of microservices. But in this case, event-driven is slightly similar to message. The application scenario of the message often occurs within an organization, and the producer of the message knows how the message will be processed. For example, in a team, the producer and sender of the message may be the same team and the same business, and there is a very strong agreement on the content of the message. In contrast, events are more loosely coupled. For example, the event sender does not know where the event will be delivered, who will consume it, who is interested in him, and has no expectations about how the event will be handled. Therefore, the event-based architecture is more decoupled. The application of messaging is often still inseparable from the same business department, even if some large companies involve cross-department cooperation at most. The use of messages is constrained by documents, and events are constrained by Schema, so we believe that events are more completely decoupled than messages.

Next, what is the difference between the microservice architecture and the EDA architecture?

The first is the microservice architecture, which is an architecture evolved from a single application. For example, a single application is divided into many microservices, and the microservices are organized and connected through RPC. In the past, a business might arrange a bunch of functions locally, but now they are connected by a bunch of RPCs. For example, if a user places a front-end order operation, there may be several microservices in the background for order operation, one microservice to create an order, one microservice to process the order, and another microservice to process the order after processing. The completed message is notified. This is a typical RPC architecture.

But the pure RPC architecture has many problems. For example, all business logic is coupled together, but local method calls are replaced with remote calls. When the business growth rate reaches a certain stage, you will find that the capacity between the various microservices may not be equal. For example, SMS notifications can be completed asynchronously, but completed synchronously. This leads to how much traffic there is on the front end, and SMS notifications also need to be prepared for the same scale of traffic. When the preparation resources are not sufficient and the upstream and downstream flows are not equal, it may cause a certain micro-service to be suspended, which affects the upstream and thus produces an avalanche effect.

In this case, everyone generally introduces a message queue for asynchronous decoupling. This architecture is very close to the event-driven architecture. Let's take the user front-end to create an order as an example. The event created by the order will be sent to the event bus, event broker, and event bus. Different downstream subscribers listen to this event. deal with.

The difference is that message subscribers do message processing based on the SDK provided by message middleware vendors. Businesses often need to be modified and will also be bound by the technology stack provided by vendors. In the event-driven architecture, subscribers are generalized subscriptions, that is, they do not. What kind of technology stack the subscriber is required to develop based on, it can be an HTTP gateway, it can also be a function, or even a legacy inventory system. As long as the event broker is compatible with the business protocol, events can be pushed to different subscribers. It can be seen that generalized subscriptions are more widely used, more decoupled, and the cost of transformation is the lowest.

Alibaba Cloud EventBridge: Event-driven architecture practice

Gartner had predicted that the EDA architecture will become the mainstream of microservices in the future. In 2022, it will become 60% of the new digital business solutions, and 50% of business organizations will participate in it.

At the same time, the CNCF Foundation also proposed the CloudEvents specification, which aims to use a unified specification format to declare event communication. EventBridge also follows this standard. As a community standard, CloudEvents relieves everyone's worries about vendor lock-in and improves the interoperability between various systems, which is equivalent to agreeing on a unified language for each system. This is a very critical step.

Incidents have a unified specification in the open source community, but on the cloud, many users have purchased many cloud products from cloud vendors. These cloud products may have hundreds of millions of events generated every day, and these events lie in the logs of different cloud services. , In the internal implementation. Users can't see and don't know what happens to the cloud product instance on the cloud. Different manufacturers have different definitions of events, and there is no standard of the same kind as a whole. The incidents between the various cloud services are isolated, that is, they are not connected, which is not conducive to mining the value of the incident. There are similar problems when using open source products. Users often do not have a unified standard for data exchange, and they need to pay secondary development costs when they want to open up these ecosystems.

Finally, the status quo of event-driven applications in many scenarios is off-line, and now relatively few people use EDA architecture for online scenarios. On the one hand, because there is no event-type middleware infrastructure, it is difficult to obtain an event in real time, and while being pushed in real time, the entire link can be tracked by the business side. Therefore, the above is also the background of why Alibaba Cloud made this product.

Therefore, we have defined EventBridge, which has several core values:

1. Unified event hub: unified event interface, defines event standards, and breaks cloud product event islands.

2. Event-driven engine: massive event source, millisecond trigger capability, accelerate the upgrade of EDA/Serverless architecture.

3. Openness and integration: provides a wealth of cross-product and cross-platform connection capabilities to promote the integration of cloud products, applications, and SaaS services.

First of all, the basic model of EventBridge, EventBridge has four parts. The first part is the event source, which includes cloud service events, custom applications, SaaS applications, and self-built data platforms.

The second part is the event bus. This is the storage entity. When the event comes, it must exist somewhere for asynchronous decoupling. Similar to the topic concept in RocketMQ, it has a certain amount of storage and at the same time provides asynchronous capabilities. There are two types of event buses. One is the default event bus, which is used to collect events from all cloud products. The other custom event bus is for users to manage, define, and send and receive events by themselves, which is used to practice the EDA architecture concept. The third part is the rules. The rules are similar to RocketMQ consumers and subscriptions, but we give the rules more computing power including filtering and conversion. The fourth part is the event target, which is the subscriber. If you are interested in an event, create a rule to associate the event, which includes function computing, message service, HTTP gateway, and so on.

Here is a specific talk about this event rule. Although it is similar to a subscription, the event rule has lightweight event processing capabilities. For example, when using a message, you may need to get the message locally before deciding whether to consume it. But based on rules, this message can be processed on the server side.

Event rules support very complex event pattern filtering, including matching of specified values, such as prefix matching, suffix matching, numeric matching, array matching, and even combining these rules to form complex logical matching capabilities.

The other is the converter capability, the generalized definition of event targets, and the accepted event formats may be many, but not necessarily downstream services. For example, if you want to push an event to Dingding, the Dingding API has been written and only accepts a fixed format. Then, to push the event over, the event needs to be converted. We provide:

• complete events: does not do conversion, and directly delivers native CloudEvents.

• Part of the event: uses JsonPath syntax to extract part of the content from CloudEvents and deliver it to the event target.

• constant: The event only functions as a trigger, and the delivery content is constant.

• template converter: flexibly renders customized content and delivers it to the event template by defining a template.

• function: performs custom function processing on the event by specifying the processing function, and delivers the return value to the event target.

At present, EventBridge integrates more than 80 cloud products, about 800 event types, and immediately opened up the messaging ecosystem. For example, RocketMQ is a microservice ecosystem. If we practice the message event concept, we can directly deliver RocketMQ events. To EventBridge, these messages are processed through an event-driven architecture, and even message ecosystems such as MQTT and KafKa are integrated. In addition to getting through Alibaba Cloud's messaging products, some open source self-built messaging systems will also be opened up in the next step. Another ecology is the ISV ecology. Why does ISV need EventBridge? Take DingTalk 6.0 as an example, which recently released its connector capabilities. A lot of software needs to be installed in DingTalk. These software may be provided by the official, or may be provided by ISVs or third-party developers. This results in poor data interoperability. Therefore, we provide this capability to circulate ISV data. The last is the event-driven ecology. We are currently able to reach more than 10 event goals, and we are also continuing to enrich them.

As events are more decoupled and discrete than messages, event governance is also more difficult. Therefore, we created an event center and provided three capabilities:

• event tracking: can have a complete tracking of each event, where it was generated, when it was delivered, when was it filtered, when was it delivered to a certain target, and when was it processed successfully. Make the entire life cycle fully tracked.

• event insight & analysis: allows users to change from an EDA programming perspective to a user perspective, allowing users to more quickly understand what events are in EventBridge and perform visual analysis. EB is used to perform nearby calculation and analysis, directly import business messages into the event bus, and analyze the messages in time.

• event market: guides cloud products to define business events for cloud products, makes cloud products more open, and provides market capabilities.

Build an Alibaba Cloud unified event hub based on the RocketMQ kernel

EventBridge was built on cloud-native container services from the beginning. On top of this, first is the RocketMQ kernel. The kernel plays two roles in this product. One is event storage, which is used as storage; the other is to use subscription capabilities to convert subscriptions into generalized subscriptions. Above the RocketMQ kernel is the connect cluster. The more important capability of EventBridge is connection, so EventBridge must first have the capability of Source, source the event, and then save it; its core is the Connect cluster, and each Connect cluster has many workers. Each worker is responsible for many things, including event intake, event filtering, event conversion, event playback, event tracking, etc. At the same time, there is a Connect control plane on the Connect cluster to complete cluster management and worker scheduling.

At the upper level is the API Server, an event gateway, and EventBridge. In the world of EventBridge, there are two ways to ingest events. One is to actively source events through the Source Connector of Connect, and the other is for users or the cloud. Products can deliver events through API server and our SDK. There are many delivery methods, including OpenAPI, multi-language official SDK, and at the same time considering CloudEvents has community standards, EventBridge is also fully compatible with the community's open source SDK, and users can also post events through Webhooks.

The advantages of this architecture are very obvious:

(1) Reduce user development costs

• Users do not need additional development for event handling

• Write rules to filter and transform events

(2) Native CloudEvents supports

• Embrace the CNCF community and seamlessly connect with the community SDK

• Standard agreement unifies Alibaba Cloud event specifications

(3) Event Schema supports

• Support event schema automatic detection and verification

• Schema binding of Source and Target

(4) Any intercommunication of global events

• Established a cross-regional and cross-account event network

• Support cross-cloud and cross-data center event routing

A new trend in the cloud-native era: Serverless+ event-driven

We believe that serverless plus event-driven is a new research and development method, and each manufacturer has its own emphasis on serverless understanding, but the way to implement it is the same.

First of all, the serverless infrastructure shields the underlying IaaS, and the upper serverless runtime is computing hosting, and hosting is not only microservice applications, K8s containers, but not just functions.

EventBridge first connects this driven event source to trigger these runtimes. Because what Serverless needs most is the driver, and event-driven brings him such a capability, that is, the computing portal. When EventBridge drives Serverless to run, it then connects to the back-end service. At present, the scenarios where EventBridge and Serverless are combined are mainly loosely coupled scenarios, such as front-end applications, SaaS service provider applets, and audio and video codec and other landing scenarios.

So, what exactly is the serverless EDA architecture development model? Take function calculation as an example. First, the developer needs to transform from the application perspective to the function perspective, and implement each business logic in a function; a function represents a code fragment and a specific business. When this code is uploaded Then it becomes a function resource, and EventBridge can drive functions through events, and arrange functions to form a specific application through events.

There are many things that need to be done in the function. Everyone knows that the function has many drawbacks. The most criticized one is the cold start. Because Serverless requires scale to zero pay-as-you-go, when there is no request and no event to trigger, it should receive 0 directly, from 0 to 1 is a cold start. This cold start may take seconds to wait, because it may involve downloading code, downloading images, namespace construction, storage mounting, root mounting, there are many things in it, and various cloud vendors have invested a lot of effort in optimizing this piece. The price advantage of Serverless is obvious. It has a particularly high resource utilization rate. Because it pays according to the amount, it can achieve nearly 100% resource utilization, and there is no need to do capacity planning.

A simple example is the minimalist programming paradigm based on Serverless plus EDA. Another specific example is that the EDA architecture transforms this business in the new retail scenario. First of all, there are several key resources in the business, which may include API gateways and function calculations. First, you can open up some data, open rds and synchronize rds data, compatible with some historical architectures, and trigger computing resources, functions, and gateways at the same time . The advantages of the entire architecture are very obvious, so it has the ultimate flexibility and does not need to reserve resources.

Event-driven future outlook

We believe that there are two parts to an event-driven future. The first is to do a good job in connection, and do a good job in cloud and cross-cloud integration, so that the user's multiple structure is more efficient. The second is the integration of the open source ecosystem. We can see that the open source ecosystem is becoming more and more prosperous, so we also need to integrate the data in these open source ecosystems. In addition, there are traditional IDC computing capabilities, edge computing capabilities, and other ecosystems that require connectivity software to connect them.

EventBridge is a new computing driving force in the cloud-native era. These data can drive the computing power of the cloud and create more business value.

Recommended in the past

Based on the message queue RocketMQ large distributed application cloud best practice

Cloud message queue RocketMQ 5.0 new upgrade: message, event, stream integration processing platform

EDA event-driven architecture and EventBridge Two and three things

Yunqi Released｜A new upgrade of the enterprise-level Internet architecture to help digital innovation

Copyright Statement: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users, and the copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.