Cutting-edge sharing｜Alibaba Cloud senior technical expert Wei Chuangxian: AnalyticDB PostgreSQL annual new version released

Introduction to This article is about the sharing of "AnalyticDB PostgreSQL Annual New Version Release" by Alibaba Cloud senior technical expert Wei Chuangxian in the 2021 Cloud Home Conference-Cloud Native Data Warehouse AnalyticDB Technology and Practice Summit Sub-forum.

This article will introduce the annual new release of AnalyticDB PG through three parts.

1. AnalyticDB PG cloud native architecture

2. Analysis of core technologies of cloud native architecture

3. Evolution roadmap

1. AnalyticDB PG cloud native architecture

Alibaba Cloud's self-developed high-performance, massively extended data warehouse services, compatible with part of the Oracle/Teradata syntax ecology, are widely used in Alibaba Group's internal e-commerce, logistics, entertainment, advertising and other business departments, serving Alibaba Cloud's finance, government, enterprises, Users in various industries such as the Internet support the rapid construction of a new generation of cloud-based data warehouse services.

It has the following four characteristics: First, PB-level data response in seconds. Using vectorized computing, column storage and intelligent indexing, it leads the performance of traditional database engines by 3x times. A new generation of SQL optimizer, to achieve complex analysis statements without tuning. Second, it is stable and reliable to simplify operation and maintenance. The Feitian platform is built based on Alibaba's years of experience in building large-scale cluster systems, with intelligent hardware management, fault monitoring, diagnosis and self-recovery, and support for MPP databases to achieve high reliability and self-operation and maintenance of complex cluster systems. Third, high SQL compatibility. It supports SQL 2003, is partially compatible with Oracle syntax, supports PL/SQL stored procedures, OLAP window functions, views, etc., has complete functions and ecology, and can realize rapid application adaptation and migration. Fourth, data multimodal analysis. Support geographic information data analysis through PostGIS plug-in, built-in 100+ machine learning algorithm library, realize data intelligent exploration. Support high-dimensional vector retrieval algorithm, realize video/image retrieval with image search function.

Why should we upgrade the cloud native architecture? Since the 1980s, the database has gradually evolved from a stand-alone machine to a cloud-native architecture. In the 1980s, the database adopted a single-point database service architecture coupled with storage and computing. Beginning in the 1990s, through the ability to share storage, one storage and multiple calculations have been achieved. As computing nodes increase linearly, its storage gradually becomes a bottleneck. After 2000, with the development of big data, the data level was cut into multiple slices, and each node was responsible for the calculation and storage of one slice of data. Beginning in 2010, with the rapid development of cloud computing, databases began to evolve toward cloud native.

For the business of the data warehouse, it is inherently suitable for the separation of storage and computing architecture and supports elastic scaling. First, there are peaks and valleys in the amount of data, and the amount of data has surged on certain days, and the data warehouse needs to be expanded quickly. Second, real-time analysis. We need real-time data feedback, and the data just generated can be analyzed immediately. Third, the data warehouse must provide both historical data analysis and real-time analysis, which requires the data warehouse to have good resource isolation capabilities. Fourth, the current departmental data is becoming more and more complex, and data sharing between departments is required. Our data warehouse needs to achieve a storage, multi-department sharing, to reduce the business complexity caused by data reversal between departments.

2. Analysis of the core technology of cloud native architecture

Our current ADB has a two-tier structure, the upper layer is the master node, and the bottom layer is the computing node. The elasticity of the cloud disk is used to solve the problem of storage elasticity. The main pain point of this architecture is that the computing node has a state. Once it has a state, it will face the problem of slow data migration in the process of expansion, etc., so we change the computing node from a state to a state in the new cloud-native architecture. Stateless or weak state. State includes two levels of real data and metadata. Real data is stored in shared storage, and metadata is stored in distributed KV. Storage and calculation are completely decoupled and stateless, so that you can quickly achieve second-level elasticity ability. During the development and testing process, many performance problems were found. The first problem is that after the original cloud disk or local disk is replaced with shared storage, the response performance of shared storage is one or two orders of magnitude worse than that of local disks. We use distributed multi-layer caching to solve the performance of shared storage. problem. Second, shared storage has very good throughput capacity, but the storage engine needs to adapt to this feature. Therefore, we designed the row and column cache architecture and made a lot of performance optimizations for high throughput.

For ordinary customers, the most important thing is to achieve cost reduction. Since the price of the shared storage used is an order of magnitude lower than the original cost of using local disks or cloud disks, the cost of the entire native version will drop significantly.

The cloud native architecture has four characteristics: The first one is elasticity, which can realize independent scaling of computing and storage. The second is real-time, which retains real-time capabilities and supports high-concurrency real-time writes. The third is high throughput, with good multi-dimensional analysis performance, and linear expansion. The fourth is data sharing, which can realize real-time sharing of data across instances.

First of all, we will introduce the expansion process. Assume that there are only two computing nodes at the beginning, and the data has eight shards. Before expansion, each computing node is responsible for four shards of data, the mapping relationship is stored in the metadata database, and all data is placed on shared storage. The expansion process is to change the mapping relationship from the original one node to four shards to one node to two shards. The expansion process does not require data migration, only needs to modify the metadata, and the entire process can be elastic in seconds.

High-throughput real-time writing is an important feature of real-time data warehouses. Mainly through the following three methods: 1. Batch and parallelization to improve throughput. 2. The local row storage table realizes transaction ACID. 3. Distributed cache acceleration.

Another important technical point is co-existence away from the online integration. We design a throughput-oriented storage engine for mixed storage to give full play to the high-throughput characteristics of shared storage. The mixed storage of rows and columns utilizes the orderliness of data, supports calculation pushdown, and has achieved a performance improvement of more than 10 times. At the same time, for the scenario of multi-dimensional analysis of any column query, a multi-dimensional sorting function is designed to ensure that any query of multiple dimensions can achieve a millisecond response.

ADBPG originally used the volcano computing model, and upgraded the volcano model to a vectorized model in the cloud native version. The essence of the vectorization engine is to change the original one-by-one calculation into batch calculation, and each batch of data is calculated by columnar vectorization. Compared with the volcano model, the vectorized engine has the advantages of high CPU Cache hit rate, parallel pipeline, low function call overhead, and reduced memory fragmentation. The test results show that the vectorized computing engine has a performance improvement of more than three times compared with the original volcano model.

The first evolution feature of the computing and storage separation architecture is data sharing. Metadata can be divided into system tables and visibility tables and stored in the KV system. The shared instance synchronizes metadata to the KV system, the shared instance checks the KV system in real time, gets the metadata and visibility information of the latest table, and then accesses the data in the shared storage based on the metadata, thereby realizing real-time data sharing.

The next evolution feature is fine-grained elasticity. Through the separation of computing and storage architecture introduced above, the stateless computing nodes have been realized. The next step is to subdivide the nodes into storage servicing nodes and computing nodes. Storage servicing nodes are mainly responsible for real-time data writing and caching, and computing nodes are completely stateless, thereby achieving extreme flexibility.

Three, evolution roadmap

The evolution path in the coming year. The cloud native architecture was upgraded in October to support rapid expansion and contraction. In December, the cross-instance data sharing function was launched, and the time-sharing elastic function was supported. In June next year, storage services and computing will be stateless. In October 22, it supports operator-level flexibility and automatic suspend/start functions.

Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.