Building Alibaba Cloud event-driven engine EventBridge based on RocketMQ

Introduction to Cloud event bus EventBridge As the event hub on the cloud, the core capability is connection. Whether it is online business scenarios, IoT scenarios, or big data scenarios, whether it is Alibaba Cloud, other cloud vendors, or private IDC computer rooms, we will provide a safe and reliable integration method.

Author | Liu Weng

The cloud native technology with Kubernetes as the infrastructure has completely changed our development and thinking mode. As a first-class citizen in the cloud-native field, events have become ubiquitous and are the foundation of loosely coupled and flexible cloud-native architecture systems.

As one of the 10 strategic technology trends defined by Gartner, event-driven architecture (EDA) has gradually become the mainstream technology architecture. According to Gartner's estimates, by 2022, 60% of new digital business solutions will use EDA, and EDA will account for half of the technology stacks in which commercial organizations participate.

This article will introduce events, event-driven architecture, Alibaba Cloud event-driven engine EventBridge and its capabilities in event standardization, centralization, and event-driven architecture.

Event and event-driven architecture

1. Event

Events are facts that have happened and are immutable. In contrast, the message is the original data produced by one service for the consumption or storage of another service, and the message can be modified.

The producer of the event faithfully generates and delivers the event. It does not care who, why, and how the event will be handled. The producer of the message knows who is consuming it and knows what factors are encapsulated in the message for the consumer to process.

The Broker of the event is designed to provide a fact log. The event is deleted after the timeout period. The timeout period is defined by the organization or business. The broker of the message is designed to deal with all kinds of problems. When the consumer perceives the message, the message can be deleted.

	<span class = "Lake-fontSize-1515"> Event </ span>	<span class = "Lake-fontSize-1515"> message </ span>
<span class="lake-fontsize-1515">Data</span>	<span class="lake-fontsize-1515">Facts that have occurred and are immutable</span>	<span class="lake-fontsize-1515">Raw data produced for consumption or storage</span>
<span class="lake-fontsize-1515">Producer/Consumer</span>	<span class="lake-fontsize-1515">The producer does not know who the consumer is and how to deal with it</span>	<span class="lake-fontsize-1515">The producer knows who the consumer is and how to deal with it< /span>
<span class="lake-fontsize-1515">Broker</span>	<span class="lake-f ontsize-1515">After providing the fact log timeout, the event will be deleted</span>	<span class="lake-fontsize-1515">After handling various issues and being perceived by consumers, the message will be deleted</span>

* Discrete event: describe the change of state (state) executable * Continuous events: describe the state (condition) that can be analyzed Usually, events are discrete, used to describe a change in the state of a thing, and can be executed. Consumers perform corresponding actions based on the state described by discrete events. An event can also be a part of a continuous data stream, used to describe a thing currently in a certain state. These continuous events are analyzable, and consumers can analyze certain trends and the reasons behind them based on the changes in these states. Events should be designed to be the smallest size, simplest type, and single purpose. Here we will focus on CloudEvents. CloudEvents entered the CNCF Foundation sandbox project in May 2018, and then it took just over a year to become the CNCF incubation project, and its development speed is very fast. CloudEvents will become the standard protocol for event communication between cloud services. At the same time, it should be emphasized that CloudEvents has released a number of binding specifications for message middleware. CloudEvents * Launched in December 2017 * CNCF sandbox project in May 2018 * October 2019 1.0 CNCF Incubation Project * December 2020 1.0.1 ## 2. Event-driven ### Event-driven architecture is a software architecture paradigm that revolves around the production, detection, consumption, and response of events. It provides a basic guarantee for the distribution and scalability of cloud native applications. The natural asynchronous nature of the event-driven architecture enables cloud-native applications to be designed according to DDD theory to clearly demarcate the context boundaries between services and achieve loose coupling elegantly. ### 1) Event delivery mode Let's approach event-driven and take a look at the delivery mode of events. Unlike request-driven, the two ends of event-driven are not directly connected. There are three types of event delivery modes as follows. Queue-based producer-consumer model. This is a single receiver mode used for event delivery between two services. The producer service does not care about how the consumer service handles events. Queue-based asynchronous request-callback mode. This mode is similar to request-driven request-response, which is asynchronous request-reply or request-callback, which is also used for event delivery between two services. The producer service cares about the response events subsequently produced by the consumer service. Subject-based publisher-subscriber model. This is a many-to-many model. The publisher service may produce different types of events and transmit them to different topics, and the subscriber service may subscribe to one or more topics to handle different types of events. ### 2) Event service definition mode Let us understand the service definition mode of the next event. We already know that the producer of the event does not know who the consumer is, so the format of the message cannot be pre-defined like the message. Therefore, in the event-driven architecture, a Schema Registry is needed to provide serialization basis for producers and deserialization basis for consumers. Schema is similar to the proto definition in gRPC. In the request-driven mode, the gRPC server and client will generate stub template code according to the proto definition. Then provide the template code to your own upper code to call, so as to achieve serialization and deserialization. Similarly, in the event-driven mode, after consumers obtain events, they can parse out Schema and Content (usually binary) according to the CloudEvents standard protocol, and then call the Schema Registry service through consumers to deserialize the content into events body. It can be seen that the service definition mode of the event can fully decouple the producer and consumer of the event. ## 3、EventBridge Through the above introduction, I believe you have a clearer understanding of events and event-driven concepts. Next, I will introduce EventBridge. EventBridge is a serverless event bus service that provides users with loosely coupled and distributed event-driven architecture. Event transmission and storage of EventBridge follow the CloudEvents protocol. In EventBridge, the producer of the event is called event source , the medium for transmitting and storing the event is called event bus , and the consumer of the event is called event target . The event is converted, matched, aggregated, and routed to the event target event rule EventBridge connects the two ends of event production and consumption, and uses the capabilities of cloud native infrastructure and the characteristics of serverless consumption on demand to provide users with low-code, loosely coupled, and highly available event processing capabilities. Users can realize highly responsive EDA cloud native applications with minimal investment. At the same time, EventBridge is based on a standard event protocol, which is conducive to promoting the unification of event standards for various event sources, and gradually integrating event islands into a complete event ecosystem. Therefore, EventBridge is becoming the standard paradigm for cloud-native event-driven architecture. So, how does EventBridge combine with Serverless to implement a minimalist EDA application? In the following event bus paradigm and application scenarios, I will introduce them in detail. # Event bus ## A major feature of EventBridge is the pipeline of the standard event protocol. So, let's take a look at the various components of the Alibaba Cloud event bus EventBridge that implements this pipeline capability. ### ## 1. The composition of EventBridge ### 1) Event source The event sources of Alibaba Cloud EventBridge are all-encompassing. It can be Alibaba Cloud’s various cloud products, Alibaba Cloud’s third-party SaaS services, Alibaba Cloud users’ own services, or even services from other cloud vendors, edge services, and private computer rooms. Users can use CloudEvents SDK to push events to the Alibaba Cloud event bus, thereby realizing events to the cloud. ### 2) Event bus In order to provide out-of-the-box cloud product event processing capabilities, Alibaba Cloud EventBridge provides each user with a tenant-isolated default event bus. Events generated by cloud products used by users will be transmitted and stored by this event bus. Users can dock various event sources through a custom event bus, and collect, store, and respond to events from different data sources in a unified manner. ### 3) Event rules The two ends of EventBridge's event rules are the event bus and the event target. By configuring matching rules, conversion rules, etc., users can implement event filtering, conversion, and routing from the event bus to the event target in a low-code or even no-code manner. ### 4) Event target The event target is where the event is ultimately processed. Alibaba Cloud EventBridge currently supports a variety of event targets, bringing users an out-of-the-box experience. We can specify the DingTalk robot for an alarm event, can transmit an order event to the user service through the HTTP gateway, or post the event to the message service to implement the event on the cloud. Of course, the classic event goal of cloud native is to be served by serverless, because serverless services fully demonstrate the advantages of cloud native. * Serverless resources are consumed on demand, maximizing flexibility * Lightweight functions have low latency, high availability, and no operation and maintenance costs * The learning threshold for users to write event processing functions is low, and the amount of development code is small #### ### 5) Give an example I will show you a small example here, let's experience the lightweight of EDA realized by EventBridge+Serverless. First, we customize an event bus, using the Kubernenets container service as the event source, and the Serverless service (function computing) as the event target. Then we define an event rule for the resource state change event in the container. When this type of event enters the event bus, it will be routed to the function computing service. Finally, we write and deploy a function to handle this type of event to the function computing service. In the function, first receive the event of the CloudEvents standard protocol, parse the event through the Schema Registry, and finally complete the event processing by the function itself-such as calling the container service API to perform related operations on resources. From this small example, we can see that EventBridge+Serverless allows users to quickly implement an event-driven business at a very low cost. Four or two strokes a thousand catties. Next, I will introduce the two event-driven architecture orchestration modes, and give corresponding examples. I hope to stimulate your brain and combine your own business to get the best practice of event bus in accordance with local conditions. ## 2. Orchestration mode of event-driven architecture #### ### 1) Mediator mode For handling more complex event-driven scenes, the mediator mode can help us to disassemble and analyze the event in an orderly manner, and finally execute the specified action. The mediator model consists of three roles: external service, mediator, and executor. First, the external service acts as an event source of the event bus, and transmits the event to the event bus. As the mediator 16194a170e7bcd, the or function is the event target of this event bus, receiving and processing events from an event source. During the execution of the mediator function, multiple intermediate states of event processing will be used as new events and transmitted to the corresponding event bus. At this time, the mediator is the event source of these event buses. The function of as the executor of is the event target of these event buses. These functions receive and process events from the event bus, and then generate a callback event and transmit it to the corresponding event bus. It can be seen that between the mediator and the executor here is the asynchronous request-callback model mentioned earlier. After receiving the callback event, the mediator executes the mediation logic, and uses the result as a callback event to transmit to the external service via the event bus. ### 2) Example: Smart Home Next, let me show an example of using the mediator mode to implement a smart home. In this example, we use IoT devices/sensors as and services , the user's whole house system as mediator , and the function created by the user in the function calculation as First, the sensor generates an "air quality exceeding standard" event and transmits it to the user-defined "air quality" event bus. After the user's whole house system receives this event, it calculates the indoor and outdoor air quality separately, and finds that the outdoor air exceeds the standard, and then sends the air quality event inside and outside the window to the user-defined "air inside and outside the window" event bus, and the window control function is used as the execution Or, issue a window closing command to the IoT service, and then transmit the window status event. After the whole house system knows that the windows are closed, it continues to send corresponding events to the corresponding event bus such as fresh air control and light control according to the user-defined whole house logic to complete the whole house control. The example of the mediator pattern is introduced here. Next, I will introduce the pipeline and filter mode. ### 3) Pipe and filter mode The pipeline and filter pattern consists of three roles: source service, pipeline function, and target service. The events generated by the source service go through multiple event buses and are converted, filtered, aggregated and other operations performed by the corresponding pipeline functions. Finally, the new events are transmitted to the target service via the event bus. ### 4) Example: online learning Next, let me show an example of using the pipeline and filter pattern to realize the online learning of artificial intelligence services. We all know that the boom of AI originated from big data. All kinds of artificial intelligence services we use today have one or more business algorithm models behind them. These models are usually trained repeatedly by algorithm architecture + offline, batches of big data. Because these services have natural data relevance, real-time online data will help the improvement of the model to a certain extent. Here, we assume that the travel recommendation system is the source service , the travel model training service is standard service , and the functions in the middle are pipeline functions . The travel recommendation system transmits real-time road conditions to the real-time traffic event bus. The event target of the real-time traffic event bus is two functions with different functions. * The first function is responsible for completing data cleaning and feature extraction, then generating feature events and transmitting them to the feature event bus. * The second function is responsible for data labeling. After marking the original data, a labeling data event is generated and transmitted to the labeling data event bus. The event target of the characteristic data event bus is also two functions with different functions. No more details here. Finally, the travel model training service will train a new model based on real-time data. The details of the industrialization process such as model regression are omitted here. # Event Center ## At this point, we have a certain understanding of EventBridge as a pipeline for event transmission, storage and routing. Next, I will introduce another major feature of EventBridge, the centralized query, display, and analysis capabilities of events. As mentioned earlier, EventBridge is based on the standard event protocol CloudEvents, and is gradually unifying and integrating event islands to form a complete event ecosystem. Under this ecosystem, the event center unifies the events generated by users' cloud products, third-party SaaS services, user services, and services under the cloud, and provides users with a full range of event query, visualization, and analysis capabilities. * The event center provides users with query and analysis of one or more event buses in the tenant dimension. * For services or cloud products with distinctive event characteristics, the event center provides a market and charts to facilitate users' real-time observation. * At the same time, the event center provides event alarms and event cards, enabling users to handle specific events with zero codes. The event center maximizes the value of events and provides users with traceability, measurability, and observability for cloud-native services from the perspective of events. # Outlook ## As the event hub on the cloud, the core capability of Alibaba Cloud event bus EventBridge is connection. Whether it is online business scenarios, IoT scenarios, or big data scenarios, whether it is Alibaba Cloud, other cloud vendors, or private IDC computer rooms, we will provide a safe and reliable integration method. In the future, EventBridge will focus on the development of ecological networks. In the cloud era, such a huge nerve center system cannot be built in a day. We need and look forward to working with you to build a cloud-native event-driven architecture ecosystem. ﹀ ﹀ ﹀ group number 16194a170e8319 31704055 to join the technical exchange group, Get detailed cloud native solution information and expert answers. > Copyright Statement: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.