Feng Zhongyan: The Future of OceanBase Community Edition 4.0

On August 10th, the 2022 OceanBase annual conference was held in Beijing, Shanghai and Shenzhen at the same time. Feng Zhongyan, senior open source ecological technology director of OceanBase, shared the keynote speech of "OceanBase Community Edition 4.0 Future Imagination" at the meeting, and shared the OceanBase community with everyone. New planning and vision for version 4.0.

The following is the transcript of the speech:

Hello everyone, I am in charge of open source related work at OceanBase. Today I am very happy to share with you the future of OceanBase Community Edition 4.0. Before sharing, I will introduce a little episode. In the past one to two months, many small partners in the community often ask questions in the community, whether certain functions will appear in version 4.1 or which version will appear in the future, when 4.1 will be released, etc. .

Today, I am honored to take this opportunity to officially synchronize with you: our OceanBase Community Edition 4.1 will meet you at the end of this year!

My sharing today is divided into four parts, first, about ecology; second, about what kind of community we want to build; third, about the features of 3.1.4; fourth, what the future 4.1 will look like. In the past year, many friends in the community have asked what our community has developed? I would like to sum up in this sentence "open source and open source, ecological win-win".

Open source and open, ecological win-win

In the past year, we have been continuously connecting with ecological tools to promote the common development of the ecology. When we started open source on June 1 last year, we only provided a 3 million-line kernel plus an offline synchronization tool. We are very grateful to the first batch of users for their support. During the process of using the first version of OceanBase, we found a lot of Problems, such as real-time data synchronization, such as monitoring, so we quickly docked K8S and Prometheus in version 3.1 to make real-time monitoring and installation and deployment smoother.

However, with the increasing number of users, we found that users need a complete ecological chain, so in version 3.1.2, we have opened up the OMS, OCP, ODC that we have developed in the commercial version for many years, so that users can have A complete experience. Version 3.1.2 is a very significant release in open source because it makes it easier for users to use OceanBase. In versions 3.1.3 and 3.1.4, we continued to connect and connect with ecological tools. For example, we made a connection with Mr. Bai's intelligent diagnosis platform and operation and maintenance management platform D-SMART. Last month, we also made a connection with Navicat. A joint announcement was made announcing support for Navicat in OceanBase Community Edition.

Community People often ask what kind of community do we want to build? The answer is, we want to build a responsive community. This page of PPT shows all the core data of the community. We can pay attention to this curve. This curve is a graph of the daily message volume of Dingding in the past month. The average daily message volume is about 300. The other data is the Q&A community. The data of the Q&A community is more than 14,000 messages. From the open source last year to now, there are more than 1,000 Q&A messages per month on average. Measured from these indicators: At present, our community activity is in a very healthy state.

Over the past year, there have been 35 technical sermons, an average of 2.5 per month. Another interesting thing is that last year we held the first OceanBase database competition, which is a competition for developers. There are nearly 2,000 students or database hobbyists from 247 famous universities such as Tsinghua University, Peking University, Renmin University of China, Nanyang Technological University, and Columbia. to participate in the competition. We plan to officially launch this year's OceanBase database competition on the 12th of this month. The community is constantly communicating with customers, and we look forward to becoming friends with customers.

1-3-1, what is asked is what is answered, and timely response

The number "1-3-1" means that we expect that any inquiries from users can be answered within an hour in the DingTalk group and work log, and within 3 hours in our Q&A forum, and in Github Responses are available in the last day, and we want to be a community that responds in a timely manner.

We continue to communicate with users and listen to their voices. We also want to thank the community members, such as Ah Fu, Ma Shunhua, Yan Shaoan, and community stars that cannot be listed one by one.

It is very difficult to launch an open source project at the beginning, and it is an honor to have more than 200 users trust us. To date, we have more than 100 community contributors to thank for their support and dedication. In particular, I would like to thank the customers who trust us, including financial companies UnionPay, Sunshine Insurance, Puyin AXA, Internet 360, Kuaishou, gitee, and e-commerce companies JD.com, Zhiou Furniture, and life services. 58, Yuanwang, the new pulse of enterprise services, etc.

The Community Edition has three very typical scenarios. The first typical scenario is cost reduction and efficiency enhancement. In Ctrip's scenario, our data is greatly compressed, many low-load applications are merged into a cluster, and the multi-active and multi-availability in different places is fully reused, so that the entire cost can be reduced. Nice drop.

Another typical scenario is the upgrade of HBase. In the shell scenario, our storage cost is reduced by half, and in the secondary index scenario, the performance is improved by 6 times, and the high availability of OceanBase is fully reused, so that the The whole system runs more smoothly.

The third typical scenario is the mixed load of HTAP. In China Telecom's scenario, the performance of our complex query is improved by 5 times compared to the previous system, and the amount of data we compress is 1/3 of the original, we guarantee OLTP and OLAP mixed load.

Community Edition 3.1.4 Interpretation

We released 3.1.4 last month, and today I will give a detailed introduction to the more important features of this version. In the past year, we can use this sentence to describe the iteration of OceanBase's open source version, which is "a fast-running integrated HTAP database".

In our 3.1.0, the kernel is completely open; in versions 3.1.1, 3.1.2, and 3.1.3, we fully embrace the open source ecosystem and open up our capabilities, especially like OCP, OMS, ODC; in 3.1.4 And in 3.1.5 we have greatly improved usability and improved the stability of the kernel.

A little bit about a special feature - diagnostic capabilities. Before 3.1.4 of the Community Edition, if the user uses improper SQL, improper database configuration, or improper application usage, and the traffic is very large, some exceptions will occur in OceanBase. At this time, we mainly rely on experienced DBAs for fault diagnosis. , viewing logs and viewing monitoring indicators, which is very tedious and painful. In version 3.1.4, our OCP has opened the fault diagnosis capability, which can automatically identify slow SQL and suspicious SQL. When slow SQL and suspicious SQL appear, the system will alarm, and DBA will limit the slow SQL flow after receiving the alarm. , so as to make the whole system run smoothly and ensure the stability of the system. Even sometimes, if the DBA finds that the execution plan of some SQLs in the whole system is not optimal, which brings some risks to the system, you can use the outline to modify the system and use a more efficient execution plan. This is also the fundamental ability of one DBA of MYbank to operate and maintain more than a few thousand clusters as just shared.

Before the 3.1.4 community edition, there was another pain point called backup and restore. If we want to do a backup or restore, we need to enter a very long command. There are a lot of parameters in this command. These parameters may cause some inexplicable errors if they are entered carelessly. Frankly speaking, it is very painful. 3.1.4 In response to this pain point, we provide OCP to directly support physical backup and recovery, so that the whole support is very simple and easy.

Another major pain point is remote multi-activity and remote disaster recovery. In the past versions of the community edition, if we wanted to do multiple activities or remote synchronization in different places, we needed to configure a very complex open source tool to support the synchronization between clusters. In version 3.1.4, the OMS community edition opened up between OceanBase and OceanBase versions. For real-time synchronization, we provide asynchronous synchronization to support remote multi-active and remote disaster recovery.

4.X's future imagination

After a lot of foreshadowing to introduce the state of the past, we began to imagine the exciting future of 4.X. This morning, Mr. Yang Zhenkun made a lot of interpretations of 4.0, and I will give a detailed introduction to the 4.X of the future community version.

One of the biggest features of 4.0 is: our open capabilities are completely flush with the commercial version. On the MySQL series, it can better meet business demands and support more business scenarios; in terms of core capabilities, it is cost-effective and has small specifications. , There is a stand-alone improvement, and there is an improvement in the ability of HTAP.

Regarding high availability, there are three particularly big features. The first is that the RTO is increased to 8 seconds; the second is IOPS isolation; the third is the active and standby databases. IOPS isolation is a very good feature. In many user scenarios, they can combine many low-load applications, thereby greatly improving the utilization of the entire cluster, which is very helpful for reducing costs. In addition, 4.0 will support MySQL 8.0 in terms of compatibility; there are many good features in multi-function, such as cross-database query, we will support DBLink, and DDL will support functions that were often wanted in 3.X before, such as primary keys Changes, type conversions, etc.

There is also a lot of demands for partition management, which will be introduced in detail later. We have a very big improvement in multimode, and we support Gis. In the exhibition area, we have a very cool Raspberry Pi. Its specification is a 4-core 8G architecture. On such a low configuration, OceanBase 4.0 can still run smoothly, which means that in the development environment we used to go from 8-core 64G Down to 4 cores 8G. In the future, we will do further optimization. In the stand-alone environment, our OceanBase will surpass MySQL in the OLTP scenario. This is the result of TPC-H 100G. The 4.0 version has nearly 5 times the performance improvement compared to the 3.X version. It is an overall improvement rather than an increase in the amount of SQL that was often seen in the past. It is an overall improvement in all SQL. .

What is the weapon behind the performance improvement? The main reason is: in version 4.0, we will also open up the mixed storage of rows and columns, and open up the vectorized computing engine, and the entire optimizer will be greatly improved. In the past year, we have often received a request - automatic management of partitions. The design of partitions is a test of DBA skills. If the partitions are well designed, the overall performance will be very good, but if the partitions are not well designed, the overall performance There will be a cliff-like drop. Starting from 4.1, we will support automatic management of partitions. As the amount of data continues to grow, our partitions can be automatically split. If the amount of data shrinks, we can merge multiple partitions together, which greatly releases the pressure of DBA and allows DBA to Partition management has become easier, and OceanBase can be better used.

The new online experience center we launched last month has a very good feature, it adds a help center, I will give a simple example, suppose I want to do a create table operation, click "insert into workspace" , a create table template will be created in the workspace. The whole process is very simple and smooth, DBA does not need to remember complicated instructions, and the applicability is very convenient, allowing DBA to complete the learning in the experience. Last month, the official website and Q&A forum of the community were refactored and upgraded. The official website of our entire community is divided into four sections: learning, activities, content, and Q&A. The four sections better support community users to participate in interaction and experience OceanBase, and enable community users to truly become the protagonists of the community. We hope to create a A community where you can learn easily and use it with confidence.

Finally, thank you for listening. We want to synchronize an event. From tomorrow, we will officially launch a training of "From 0 to 1 Practical Database Kernel Tutorial" . This event is free. We are aimed at developers, and the expected goal is to help development Developers and database enthusiasts can get started with database kernel development without foundation. We will introduce a lot of practical database kernel development skills, such as the establishment of database development environment, introduction of database modules, unit testing, etc. I hope everyone can participate. , Let's enjoy the joy of learning together! This concludes my sharing today, thank you all!