Flink CDC + OceanBase Full Incremental Integrated Data Integration Solution

This article is compiled from the speech delivered by OceanBase technical expert Wang He (Sichuan Fan) at the Flink CDC Meetup on May 21. The main contents include:

1. Introduction to OceanBase
2. Implementation principle of Flink CDC OceanBase Connector
3. Flink CDC + OceanBase Application Scenario
4. Flink CDC OceanBase Connector Future Prospects

Click to view live replay & speech PDF

1. Introduction to OceanBase

OceanBase is a distributed database developed by Ant Group. Since 10 years, the project has been established and developed and iterated. The earliest user is Taobao's favorites. In 2014, the OceanBase R&D team migrated from Taobao to Ant Group, mainly responsible for supporting Alipay's internal de-IOE work, that is, replacing the Oracle database used by Alipay. At present, all Ant Group databases have been migrated to OceanBase. On June 1, 2021, OceanBase was officially open-sourced and a MySQL-compatible version was opened.

The OceanBase database has undergone three generations of architecture upgrades, from the distributed storage system initially used in e-commerce, to the later general distributed database, to today's enterprise-level distributed database.

The above figure shows the architecture of OceanBase.

The top-level App accesses the server side of the OceanBase database through OBProxy (load balancing proxy). There are multiple copies of the data on the server side. The relationship between the copies is similar to the master-slave relationship in the database architecture, but it is at the table level, that is, the partition The partition of the table is that there are multiple copies of the table level, and then scattered in multiple servers.

The architecture of OceanBase has the following characteristics:

• Shared-nothing architecture : Each node has its own complete SQL engine, storage engine, and transaction processing logic. Nodes are completely equal, and there is no hierarchical structure.
• Partition-level availability : Provides partition-level availability. In the OceanBase database, partition is the basic unit of reliability and scalability, which realizes access routing, load balancing and automatic failure recovery.
• High availability + strong consistency : Since there are multiple copies of data, the Paxos consistency protocol is used between multiple copies to provide high reliability, and to ensure that log persistence is successful on the majority node.

The core features of OceanBase are as follows:

1. High availability : Based on Paxos protocol, strong consistency. A few replicas fail, data is not lost, and services are kept non-stop.
2. High expansion : supports online horizontal expansion and shrinking, and can automatically achieve load balancing between nodes.
3. High Compatibility : Community Edition provides compatibility with MySQL protocol and syntax.
4. Low cost : The usage cost of OceanBase database storage is about 1/3 of that of MySQL. Because it has lower requirements for hardware quality and a lot of optimizations for storage, the "storage compression ratio" is extreme.
5. Multi-tenancy : The resources between tenants are completely isolated, and different business parties only need to manage data in their own tenants, which can save a certain amount of cost.
6. HTAP : It implements the functions of OLTP and OLAP simultaneously in a set of engines.

2. Implementation principle of Flink CDC OceanBase Connector

The current mainstream CDC implementation method mainly relies on the database log. After obtaining the incremental log of the database, it is necessary to ensure its orderliness and integrity, and then process these logs, and then write them to the destination, such as data. warehouse or query engine.

OceanBase provides some components for the acquisition of incremental data. Because it is a distributed database, its data is also scattered when it falls into the log. It provides an obcdc component for getting database logs. It interacts with the OceanBase server through RPC and pulls the original log information. After certain processing, an ordered log stream can be spit out, and the downstream consumes the ordered log stream by accessing the obcdc component.

At present, there are three main types of downstream consumers:

• oblogproxy : an open source component, a service that consumes log streams, Flink CDC relies on this component to implement incremental data pull.
• OMS store : Data migration service provided by OceanBase. The commercial version of OMS has iterated many versions and supports many data sources. Last year, OMS provided community edition support, mainly supporting OceanBase's community edition and MySQL two data sources.
• JNI client : The log client of JNI can directly use obcdc to interact with OBSserver to pull incremental logs, which is under open source plan.

At present, there are mainly two OceanBase CDC components provided by the open source community:

• OceanBase Canal : Canal is Alibaba's open source MySQL incremental log pulling tool. Based on the latest code of the open source version of Canal, the OceanBase community has added the ability to pull and parse OceanBase incremental logs.
• Flink CDC : Using obcdc through oblogproxy, after pulling incremental logs from OceanBase, the incremental logs are consumed and processed through another open source component logproxy-client.

The lower left corner of the above figure is the definition method of the dynamic table. The data flow will be converted into a table in the form of a dynamic table in Flink. Only by converting it into a table can SQL operations be performed on it. After that, Continuous Queries will query the growing streaming table, and the obtained data is still in the table structure, and then converted into streaming data and sent downstream.

The above figure shows the implementation principle of Flink CDC.

The Flink CDC Connector only reads data from the source, that is, it is only responsible for reading data from the data source to the Flink engine.

The current Flink CDC Connector is mainly divided into the following three categories:

• MySqlSource : Implements the latest source interface and concurrent reads.
• DebeziumSourceFunction : SourceFunction is implemented based on Debezium, and supports older versions of MySQL, Oracle, MongoDB, SqlServer, and PostgreSQL.
• OceanBaseSourceFunction : Implements the SourceFunction interface, and implements full and incremental reading based on JDBC and logproxy-cilent respectively.

The figure above shows the data path of the Flink CDC OceanBase Connector.

Incremental data is first pulled through logproxy, and logproxy-client will monitor the data stream of incremental logs. After the data stream enters Flink CDC, it is written to Flink through the processing logic of Flink CDC. Full data is pulled through JDBC.

The capabilities currently supported by the Flink CDC OceanBase Connector are mainly limited by logproxy, which currently supports pulling data from a specified time. However, since OceanBase is a distributed database, the starting point of log incremental data cannot be accurately found, and there may be some duplicate data specified by timestamp.

In the full volume stage, because OceanBase Community Edition does not have table locks, the data boundary cannot be determined by locking when reading full data.

Based on the above two considerations, only the at-least-once working mode is currently supported, and exactly-onece has not been implemented yet.

3. Flink CDC + OceanBase application scenarios

3.1 Scenario 1: Data integration based on sub-database and sub-table

Flink CDC is a full-incremental integration, and OceanBase Connector supports regular matching for reading data tables. For some scenarios of sub-database and sub-table, you can create a dynamic table through the OceanBase Connector to read the data of the data source, and then write it into a table to realize the aggregation of table data.

3.2 Scenario 2: Data integration across clusters/tenants

OceanBase is a multi-tenant system. Currently, the cross-tenant access capability has not yet been implemented for the community version of MySQL tenants. Therefore, if you need to read data across tenants, you need to connect to multiple databases to achieve separate reading. Flink CDC is naturally suitable for this work, which is equivalent to each tenant corresponding to a dynamic table as a channel for data source reading, and then aggregated in Flink.

3.3 Scenario 3: Data Integration of Multiple Data Sources

Data aggregation can be performed on different types of data sources. For the integration of MySQL, TiDB and other data sources compatible with the MySQL protocol, because the data format is the same, there is basically no modification cost.

3.4 Scenario 4: Building an OLAP application

OceanBase is an HTAP database that not only has strong TP capabilities, but can also be used as a data warehouse. The JDBC connector in Flink supports writing data to a database compatible with the MySQL protocol, so you can use Flink CDC to read the source data, and then write the data to OceanBase through the Flink JDBC connector, using OceanBase as the target. use.

At present, OceanBase provides three data usage methods: SQL, Table API and HBase API, and all the components required for the usage are open source.

Fourth, the future prospect of OceanBase Connector

The figure above lists the current status of the OceanBase CDC program.

OMS Community Edition : It is a functional subset of the OMS commercial edition, but is not open source. As a white-screen tool, it is user-friendly, fully-incremental and integrated, with data verification and O&M capabilities. Its disadvantage is that the deployment process is slightly cumbersome. It only supports MySQL and OceanBase Community Edition data sources, and does not support incremental DDL.

DataX + Canal/Otter : It is an open source solution, and data migration is done through DataX + Canal/Otter. Otter is Canal's parent project, mainly for remote multi-activity, can support two-way data synchronization, and its incremental data reading is based on Canal. The advantage of this solution is that it supports a variety of destinations, and supports HBase, ES, and relational database RDB; the disadvantage is that Canal and Otter do incremental, DataX does full, and the way of separating incremental and full is relatively separate. , there will be data redundancy in the connection part.

Flink CDC : A pure open source solution, with an active community and fast growth of community users, supporting a variety of sources and destinations, with full incremental integration. At the same time, Flink, as a very good big data processing engine, can do ETL. The disadvantage is that OceanBase Connector currently does not support incremental DDL, and does not implement exactly-once, so there may be data redundancy in the overlapping part of incremental and full.

In the future, we will optimize data reading first. To parallelize the full part, use the parallel processing framework of the new source interface; for the incremental part, skip the logproxy service and directly pull the incremental data from the OceanBase database, that is, use the obcdc component to directly pull the data through the JNI client.

Second, rich features. At present, Flink CDC only supports the community version of OceanBase, and the OceanBase community version and the enterprise version use the same components in incremental log reading, so only minor changes can support the incremental reading of the enterprise version; support incremental DDL , exactly-once mode, and speed limit.

Finally, improve code quality. First, end-to-end tests will be added; in the format conversion part, runtime converters are used instead of JdbcValueConverters to improve performance; support for the new version of the source interface (parallel processing framework) will be implemented.

Q&A

Q: What is the ease of use and stability of Flink CDC OceanBase Connector after open source?

A: In terms of ease of use, since open source one year ago, we have successively added many open source components, including support for non-open source community version generation functions such as OMS and OCP; in terms of stability, OceanBase has been widely used in Ant, and the MySQL-compatible version is also available. It has been put into large-scale applications in more than 20 enterprises including Kuaishou, so there is no need to worry about its stability.

Q: Where are the sharding information, index information and other metadata of OceanBase stored?

A: It is stored in the OB server and can be queried directly through SQL.

Click to view live replay & speech PDF

---

For more technical issues related to Flink, you can scan the code to join the community DingTalk exchange group to get the latest technical articles and community dynamics as soon as possible. Please pay attention to the public number~

Recommended activities

Alibaba Cloud's enterprise-level product based on Apache Flink - real-time computing Flink version is now open:
99 yuan to try out the Flink version of real-time computing (yearly and monthly, 10CU), and you will have the opportunity to get Flink's exclusive custom sweater; another package of 3 months and above will have a 15% discount!
Learn more about the event: https://www.aliyun.com/product/bigdata/en