ArchSummit Review: From cloud native to real-time data lake, how architecture supports business development

【Click to learn more about NetEase technology】

Under the theme of digitization, automation, and intelligence, the evolution of architecture is also accelerating. At the ArchSummit Global Architect Summit held recently, Pei Fei, senior technical expert and senior architect of NetEase Shufan, and Zhou Jinsong, senior technical expert of NetEase Shufan, shared the latest architectural practices of cloud-native business transformation and real-time data lake.

Business Architecture: "Vertical" and "Horizontal" Integration, Cloud Native Empowers Business Transformation

Cloud-native is the future of architecture, but the evolution of traditional business systems to cloud-native faces many pain points: On the business side, it is not easy to transform existing services, and it needs to face complex factors such as deployment, network, operation and maintenance management and control; on the technology side, cloud-native infrastructure The capabilities are still insufficient, and it is difficult to achieve smooth and non-perceptive migration and evolution.

To this end, NetEase Shufan proposed the practice of "vertical" and "horizontal" architecture. Pei Fei introduced that "vertical" is to solve technical problems and ensure that various businesses can be implemented. From the perspective of timeline, it can be divided into classic technology stacks (such as Spring Cloud, Java proxy traffic gateway), new technology stacks (such as service mesh, multi- Runtime), continue to promote the construction of technical capabilities; "horizontal" is to solve the problem of scenarios, to ensure the support, evolution and stability of various scenarios, one is to expand the platform's coverage of business scenarios, and to improve the support capability of complex scenarios, the other is to Solve the support problems in various stages of business architecture evolution, such as coexistence of new and old, interoperability, stability, etc.

Taking a typical microservice as an example, in terms of vertical technology, we need to start with the microservice framework and service grid respectively to solve the pain points and evolution of the microservice framework, and improve the usability and performance stability of the service grid; horizontally In terms of business, it is necessary to achieve a smooth evolution from a microservice framework to a service mesh.

Specifically, the introduction of a microservice framework faces pain points such as limited scope of application, large business intrusion, heavy burden brought by the introduction of the framework, high upgrade costs of the framework, limited governance capabilities, and conflicts with cloud-native architecture evolution. The "vertical evolution" of the NetEase Shufan microservice framework is realized through the non-intrusive Java Agent technology commonly used in application performance monitoring (APM). Integrated in a Java Agent, it realizes that the service governance capability logic does not invade the business code, and finally the business can access a full set of microservice governance capabilities such as service registration discovery, circuit breaker downgrade, current limiting, monitoring, and configuration with one click.

Another vertical technical service grid of microservices is more complex. NetEase Shufan focuses on the smooth access of services and the building of large-scale business cluster support capabilities. For the former, it builds multi-protocol support, traffic management, hot upgrade of core components, and governance. Capability enhancement and Envoy gateway and other capabilities, for the latter, the performance and stability of the data plane have been improved, and the large-scale cluster support of the control plane and multi-level disaster recovery capabilities have been realized.

In terms of "horizontal business", migrating from a micro-service framework to a service mesh requires multiple registration centers, unified control protocols, and non-intrusive Agent capabilities to solve mutual discovery and access between services, unified configuration and control of Agent and Sidecar, and architecture upgrades support, etc. NetEase Shufan proposed a platform concept of dual-engine multi-mode service governance, which is used to guide the construction of platforms related to microservice technology and architecture, and to support mutual discovery, mutual invocation, Unified governance covers Agent mode, Sidecar mode, Agent+Sidecar mode, multi-runtime mode, SDK mode and other governance modes.

Pei Fei finally summed up the "three-character formula" for the integration process of "vertical" and "horizontal" architectures. The first is "fast". Architecture decisions must be decisive and long-term. Future-oriented architecture decisions should be made, and timing should be effectively controlled. The second is "deep". It is necessary to deeply cultivate the core technology, and to be able to deeply control the various technologies involved in the evolution of the architecture, so as to adapt to changes without change. The last is "stable". Whether it is the initial state, the intermediate state, or the final state, the architecture evolution plan must be stable and evolve gradually.

Data platform: stream-batch integration, data lake integration with real-time scenarios

The data development model also faces the problems of integration and smooth evolution. The classic Lambda architecture is the right-hand man for enterprises to complete real-time big data at low cost under the background of the first mature offline link technology. However, the flaw of stream-batch segmentation eventually broke out with the increase of data scale and the deepening of business practice. Such as data silos, fragmentation of R&D systems, ambiguity of indicators and semantics, etc. Therefore, the integration of flow and batch and integration of lakes and warehouses have become new pursuits in recent years.

How to realize this architecture evolution smoothly and at low cost on the basis of the existing data platform? NetEase Shufan proposed the architectural idea of format compatibility and engine equality, and created the Streaming LakeHouse Service, that is, using a table service layer on top of the mainstream data lake table formats (Iceberg, Delta, Hudi, etc.). Bridging the data lake with each engine, supporting the integration of stream and batch, and forming the open source project Arctic ( http://github.com/NetEase/arctic ).

For Arctic, format compatibility refers to 100% compatibility with Iceberg/Hive's table format and syntax, and engine equality means supporting both Spark and Flink to read and write data, and Trino/Impala to query data. Zhou Jinsong introduced that Arctic, through the creation of two major capability sets, real-time data update and integrated streaming-batch function encapsulation, not only supports streaming-batch mixed use scenarios, but also solves a series of functional and performance problems such as lake warehouse data management and optimization. The former includes support for primary key definition, uniqueness guarantee, structure self-optimization to ensure read and write efficiency, etc. The latter includes real-time write/read interface with millisecond-level delay, minute-level delay At the Optimize level, Arctic's optimizer uses Minor Optimize and Combined with Major Optimize, the change data and the basic data are optimized respectively (the data change of the lake warehouse table is perceived by the metadata center of the triple). The batch write/read interface, Flink/Spark/Trino and other engine support, etc.

Among them, supporting the primary key definition is a key innovation of Arctic and the data lake table format. It not only helps to achieve more optimized CDC, but also the primary key uniqueness constraint is also the basis for Merge on read (merge on read) and optimize, regardless of data management. Performance guarantees are also crucial.

At the Optimize level, Arctic's optimizer combines Minor Optimize and Major Optimize to optimize change data and basic data respectively (data changes in lake warehouse tables are perceived through the metadata center of triples).

In addition, Arctic also provides functions such as file indexing, hidden queue, and multi-write consistency guarantees to ensure that this architecture is implemented in business.

Zhou Jinsong introduced that in a business push marketing analysis, through the introduction of Arctic technology, the business switched batch analysis to real-time production, realized the reuse of production processes, and optimized the report response time, which was improved by 10 to 100 times.

summary

Just like the theme of ArchSummit - "Upgrading Architecture Thinking, Supporting Business Development", regardless of business architecture or data platform, NetEase's Shufan architecture practice closely follows the needs of business development and aims to realize the value of cutting-edge technologies. , smooth evolution has become a reality. In this context, only by practicing hard and deeply understanding the essence of technology can we master the upgrading of architectural thinking and meet business needs.

【Click to understand, brand activities, real and effective👆】