Hologres Secret: High-performance native acceleration of the core principles of MaxCompute

Introduction to Hologres Technology Secret Series is continuously updated. In this issue, we will bring Hologres high-performance native acceleration query technology principle analysis of MaxCompute.

Hologres (Chinese name interactive analysis) is a one-stop real-time data warehouse self-developed by Alibaba Cloud. This cloud-native system integrates real-time services and big data analysis scenarios. It is fully compatible with the PostgreSQL protocol and is seamlessly connected to the big data ecosystem. With the same data architecture, it supports real-time write real-time query and real-time offline federated analysis at the same time. Its emergence simplifies the structure of the business, while at the same time providing real-time decision-making capabilities for the business, allowing big data to exert greater commercial value. From the birth of Alibaba Group to commercialization on the cloud, with the development of business and the evolution of technology, Hologres is also continuously optimizing its core technical competitiveness. In order to let everyone know Hologres better, we plan to continue to launch the series of demystifying the underlying technical principles of Hologres. From high-performance storage engine to high-efficiency query engine, high-throughput write to high-QPS query, etc., a comprehensive interpretation of Hologres, please continue to pay attention!

highlights:

• 2020 VLDB paper " Alibaba Hologres: A cloud-Native Service for Hybrid Serving/Analytical Processing "
• Hologres Secret: first public! Alibaba cloud native real-time data warehouse core technology revealed
• Hologres demystified: first demystified cloud native Hologres storage engine
• Hologres secret: Hologres high-efficiency distributed query engine

In this issue, we will bring an analysis of the technical principles of Hologres' high-performance native acceleration query MaxCompute.

With the continuous enrichment of data collection methods and the accumulation of large amounts of industry data, the scale of data has grown to the level of massive data (TB, PB, EB) that cannot be carried by the traditional software industry. MaxCompute (formerly known as ODPS) was born for this reason and is committed to batch structure. It provides solutions for mass data warehouses and analytical modeling services for the storage and calculation of customized data. It is a fast and fully managed EB-level data warehouse solution.

Hologres naturally seamlessly integrates with MaxCompute in offline big data scenarios. It can speed up query MaxCompute without data import and export, fully compatible access to various MaxCompute file formats, and realize millisecond-level interactive analysis of PB-level offline data. Behind all of this, SQE (S Query Engine), the executor behind Hologres, is inseparable. Through SQE, the native access to MaxCompute is realized, and then combined with the processing of high-performance distributed execution engine HQE to achieve extreme performance. .

Hologres accelerate query MaxCompute mainly has the following advantages:

• High performance: can directly speed up query of MaxCompute data, and has sub-second response query performance. It can directly query ad hoc in OLAP scenarios, satisfying most analysis scenarios such as reports.
• Low cost: After several years of development of MaxCompute, users store a large amount of data on MaxCompute and can directly access it without redundant storage; on the other hand, users can only migrate some data in high-performance scenarios to SSD Above, data of analysis scenarios such as reports can be stored in MaxCompute to further reduce costs.
• more efficient: realizes native access to MaxCompute, without the need to migrate and import data, you can access various MaxCompute file formats with high performance and full compatibility, as well as complex tables such as Hash/Range Clustered Table, and reduce user costs.

SQE architecture introduction

As shown in the figure above, it is the overall architecture of SQE. It can be seen that the overall architecture is also very simple. MaxCompute data is stored in Pangu. When Hologres executes a query to speed up the query of MaxCompute data, on the Hologres side:

• Hologres Frontend requests the SQE Master to obtain Meta and other related information through RPC.
• Hologres Blackhole requests specific data-related information from SQEExecutor through RPC.
• SQE consists of processes with two roles:
• SQE Master is responsible for processing Meta-related requests, and is mainly responsible for obtaining tables, partition metadata, authentication, and file fragmentation.
• As the core of SQE, SQE Executor is responsible for specific read data requests, involving Block Cache, pre-reading, UDF processing, expression push processing, index processing, Metric, Meter and other functions.

MaxCompute appearance engine core technology innovation

Based on the SQE architecture, the ability to achieve high-performance accelerated query of MaxCompute data is mainly based on the following technological innovation advantages:

1) Abstract distributed appearance

Combined with the distributed characteristics of MaxCompute, Hologres abstracts a distributed appearance to support access to MaxCompute distributed data. Currently, it supports access to MaxCompute distributed Pangu files across clusters and reads them as nearby as MaxCompute computing clusters.

2) Seamless interoperability with MaxCompute Meta, supporting metadata cache with version

SQE and MaxCompute's Meta are seamlessly interoperable. Meta and Data can be obtained in real time. The Import Foreign Schema command can be used to automatically synchronize MaxCompute's metadata to the Hologres external table to realize automatic creation of the external table and automatic structural update.

3) Support UDF/Expression push down

SQE implements user-defined UDF calculations by supporting UDF/expression push-down; push-down expressions can reduce the overhead caused by useless data transmission and further improve performance.

4) Asynchronous ORC Reader, asynchronous prefetch

At present, most of MaxCompute data is in ORC format. In Hologres V0.10 and above, Hologres has updated its execution engine and uses asynchronous Reader for more efficient asynchronous reading. It also supports asynchronous prefetch to further reduce reading latency; in addition, Hologres IO merge, LazyRead, and Lazy Decoding to reduce the latency of IO in the entire query and bring ultimate performance.

5) Support Block Cache

In order to avoid using IO to fetch the file every time data is read, SQE also uses BlockCache to store frequently used and recently used data in memory, reducing unnecessary IO and speeding up read performance. In the same node, the same access data can be shared with a Block Cache through consistent Hash. For example, in the Scan scenario, the performance can be improved by more than 2 times, and the query performance can be greatly improved.

6) Resident process, reducing scheduling overhead

Architectures such as traditional process models require dynamic and real-time scheduling operations such as process creation, which brings greater scheduling overhead. SQE adopts the resident process mode to avoid unnecessary scheduling overhead. In addition, it can greatly improve the hit rate and effective utilization rate of Block Cache.

7) Network Shuffle, reducing disk

Network Shuffle needs to provide a fast and stable fault tolerance mechanism. Because Network Shuffle must ensure that the sender and receiver processes are alive at the same time to complete the data shuffle. Similarly, if the traditional disk placement method is used to perform Network Shuffle Retry, although stability can be ensured, it may introduce a relatively large performance overhead due to disk IO during the Retry process. In order to solve this problem, we optimized the staged scheduling to solve the fast and stable fault tolerance problem.

Upgrade to HQE of MaxCompute appearance engine

As mentioned above, we use SQE to accelerate the query of MaxCompute external tables. The performance can be very good when querying through SQE, but there will be a layer of RPC interaction when interacting with Hologres. When the amount of data is large, the network will have a certain bottleneck.

Therefore, based on the existing capabilities of Hologres, we have optimized the execution engine in Hologres V0.10 and above. It supports the Hologres HQE query engine to directly read the MaxCompute table , which has further improved performance compared to SQE reading. There is a performance improvement of more than 30%.

This is mainly due to the following aspects:

1) It saves the interaction of RPC between SQE and Hologres, which is equivalent to saving a time of serialization and deserialization of data, and the performance is further improved.

2) Hologres Block Cache can be reused, so that there is no need to access the storage during the second query, avoiding storage IO, and directly accessing data from the memory to better speed up the query.

3) You can reuse the existing Filter pushdown capabilities to reduce the amount of data that needs to be processed.

4) Pre-reading and Cache are implemented at the bottom IO layer to further accelerate the performance of Scan.

The following is the performance data of a customer's actual online business query:

E2E runtime	SQL the Num	SQE query performance (average response)	HQE query performance (average response)	performance
2-10s	547	4,956 MS	2,609 MS	+ 47.34%
10-30s	207	16,757 MS	5,457 MS	+ 67.43%
than 30s	63 is	78,686 MS	12,666 MS	+83.90 %
Total	817	13,631 ms	4,106 ms	+69.87%

executes 817 SQL, the overall performance is improved by 70%, and the long query is improved by more than 80%. Note: This optimization is currently online in Hologres V0.10, welcome to click view the document use. # MaxCompute acceleration scene selection There are two ways to accelerate MaxCompute query in Hologres: 1) Create an external table (data is still stored in MaxCompute), the performance will be improved by 2-5 times compared to query in MaxCompute 2) Import the internal table, the performance is about 10-100 times higher than the external table The principle of creating external tables is that Foreign Data Wrappers in PostgreSQL can access data stored outside through external access interfaces. recommends You use the more convenient IMPORT FOREIGN SCHEMA method to create the appearance, which can better simplify the synchronization of metadata without paying attention to field type mapping, etc. The method of directly building external tables and combining them actually uses the optimization capabilities of the query engine to improve efficiency, but does not use the indexing capabilities of Hologres. Therefore, when importing the outer table to the inner table, the index structure of the inner table can be specified according to the query mode, and higher query performance can be brought about through these indexing capabilities. This is the reason why the external table is imported into the internal table, and the performance of the internal table is better, which can give full play to the index optimization ability of the data warehouse. The main comparisons between the two methods are as follows:

<span class = "Lake-fontSize-12 is"> Scene / dimension </ span>	<span class = "Lake-fontSize-12 is"> properties </ span>	< span class="lake-fontsize-12">Storage cost</span>	<span class="lake-fontsize-12">data volume</span>	<span class="lake- fontsize-12">Index</span>	<span class="lake-fontsize-12"> Convenience</span>
class=" Hologres internal table</span>	<span>Very good</span>	<span>High</span><span>(SSD)</span>	<span> span>	<span>Supports multiple indexes such as bitmap and clustering</span>	<span> need to import data </ span>
<span class = "Lake-fontSize-12 is"> Hologres outer </ span>	<span> preferably </ span>	< span>Low</span><span>(HDD)</span>	<span>Single Query 200GB</span><span>Limit</span>	<span>Only supports ODPS index</span> span>	<span>No need to migrate and import data</span>

It can be seen from the above comparison: * If you have a large amount of data and high performance requirements (such as within 100ms, etc.), you are sensitive to query latency, and when you have SLA requirements for the query, it is recommended that you import the data into the Hologres internal table for query access. * If it is a temporary exploratory analysis or internal business that is not sensitive to delay, you can use the MaxCompute external appearance method to reduce data movement. * In addition to the above scenarios, you can choose a suitable use scenario according to specific business conditions. # The combination of MaxCompute and Hologres The above has introduced many scenarios of how the Hologres external table query engine speeds up queries to MaxCompute, but it does not mean that all types of queries are suitable for execution on the Hologres external table engine. Hologres is a synchronous query engine designed for interactive analysis scenarios. It is oriented to scenarios where big data comes in and small data comes out. It is typically used in Serving and Analytics scenarios. MaxCompute is an asynchronous data processing engine designed for massive data processing scenarios. It is oriented to scenarios where big data is imported and big data is exported. It is typically used in ETL scenarios. In the ETL scenario, the job is submitted asynchronously, the IO interface is optimized for Scan, the calculation process requires redundant design of nodes to support high availability, and the calculation status is required to automatically retry when it fails. These are capabilities that Hologres does not have. . Therefore, MaxCompute+Hologres is combined to form a one-stop experience of data processing + service, reducing data isolation and redundancy, providing a reasonable solution architecture for big data warehouses, and supporting real-time offline integrated development experience . # summary Hologres deeply integrates with MaxCompute through SQE and makes full use of the advantages of Hologres and MaxCompute. With the goal of extreme performance, it can directly speed up the query of MaxCompute data, allowing users to conduct interactive analysis more conveniently and efficiently, and at the same time, greatly reduce the cost of analysis. , To realize the integration of offline data warehouse services. About the author: Wang Qi (Hua Qing), Alibaba technical expert, is currently engaged in the research and development of the interactive analysis engine Hologres. follow-up, we will successively launch a series of revealing the underlying principles of Hologres technology. The specific plan is as follows, so stay tuned! * Hologres revealed: first public! Alibaba Cloud Native Real-time Data Warehouse Core Technology Revealed * Hologres demystified: first demystified cloud-native Hologres storage engine * Hologres Secret: depth analysis and high efficiency distributed query engine * Hologres Secret: High-performance native acceleration MaxCompute core principles (this article) * Hologres Secret: How to realize data synchronization between MaxCompute and Hologres is a hundred times faster * Hologres Secret: __ How to Support High Throughput Upsert * Hologres Secret: How to support ultra-high QPS in online service scenarios * Hologres Secret: How to support high concurrent query * Hologres Secret: How to support the high-availability architecture * Hologres Secret: How to support resource isolation and support multiple loads * Hologres Secret: __ Vector Retrieval Engine Proxima Principle and Practice * Hologres Secret: __ Read the execution plan, query performance ten times * Hologres Secret: How to Design a Shard and Table Group in a Distributed System * Hologres Secret: How to support more Postgres ecological expansion pack * Hologres Secret: High-throughput writes into Hologres's N postures * ...... Thank you for reading, and welcome to experience Hologres, you can refer to user manual , and also welcome to scan the code to join the nail group for technical exchanges: > Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users, and the copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.