With the acceleration of digitalization, how can data be kept fresh and how can the value be realized?

The digital transformation of the global economy is the general trend. Accelerating the development of the digital economy and promoting digital transformation have become an important driving force for economic development in the next decade. Digital growth brings new opportunities. The DTC (Direct To Customer) model has become a new trend for many companies to acquire and serve end customers. However, according to reports from analysts, in the global digital transformation, only 30-40% of big data projects Success rate, enabling digital transformation through simplified basic data platforms has become a key topic.

This article is the transcript of the speech of "Insightful User Case-TiDB Industry Application Scenario Interpretation" made by Yu Jun, general manager of PingCAP Solutions Division, at the TiDB 5.0 press conference. enterprise-level distributed database product, helps enterprise users complete digital transformation .

Monetization of data value in digital transformation

In 2020, the global outbreak of the new crown epidemic occurred. Under the influence of the epidemic, various offline economic activities have accelerated the transition to online, digital transformation has accelerated, and various online service platforms, from online office, online medical treatment, online education, to online entertainment Both have experienced explosive growth. After 2020, more and more companies are considering how to further use digital capabilities to help companies complete the closed-loop business on the entire online mobile and Internet. The DTC (Direct to Customer) business model has become a trend, and more companies All have the need for data architecture transformation and upgrading.

Data processing has gone through several stages in history. Today, with the rise of digital transformation, the requirements of digital technology processing have undergone tremendous changes. These changes are mainly concentrated in the following three points:

1. Data preservation : The data is of large magnitude and requires fast processing, and at the same time, the real-time value of real-time data needs to be fully preserved;
2. end-to-end closed loop : A completely closed-loop data drive is required from the mobile end of the Internet to online enterprise-level services;
3. Data value : Under the data-driven trend, many enterprise-level applications have broken away from traditional data services and data-supported infrastructure, and quickly turned to event-driven and data-driven data monetization themes.

Throughout the process of digital transformation, we believe that data has two very core elements: First, fresh data is generated when various businesses occur. How to keep fresh data in real-time Carrying value; second, when these fresh data are generated, how to use fresh data to achieve rapid business realization in time, such as helping companies get more customers at a lower cost, how to promote corporate marketing activities more efficiently, How to complete business insights and real-time decision-making through further real-time observation and analysis of data.

TiDB enterprise-level key capabilities

Let's take a look at how TiDB uses its advanced architecture and technical support capabilities to accelerate the process of enterprise digital transformation in the process of digital transformation.

Pain points of existing solutions in the field of online and real-time processing

We call the business systems that directly undertake business transactions online as online systems. These online systems are the first stage of business generating fresh data. After years of development, great changes have taken place in data management and data usage, but the preservation and processing technology for fresh data is still at the age of 40 or 50 years. For example, it is common to use some traditional relational databases and semi-structured data processing technologies to undertake the generation and processing of fresh data. In this process, it is inevitable to use the support capabilities of different data technologies and do it in different data technology stacks. The situation of data processing in turn produced the so-called " data island " phenomenon.

In order to solve the problem of "data islands", the industry has launched a large number of scattered data technology stacks; such as data synchronization tools, data processing tools, and various long-term pipelines to solve the data connection problem between different data stacks. In this long link and complex link data processing process, although fresh data is generated, it can no longer be kept fresh in the end. At the same time, the entire processing link is too long, resulting in the loss of the real-time value bound by the data and time dimensions. On the other hand, from a business perspective, to obtain
It is also very difficult to obtain a unified view of the global fresh data under such a complex pipeline network and the interweaving situation of complex data processing links.

As an enterprise, often hopes to use various means to realize the commercial value of the data after the fresh data is generated. In other words, we will perform business-driven secondary processing on these data. Provide data consumption capabilities for different business systems. is currently in the industry, and usually has to use a very complex data technology stack complex to support. For example, online queries with high query requirements may be supported by databases such as MySQL or Oracle; for summary and detailed types, Generally, such as ClickHouse or Elasticsearch will be used to divide and conquer; including the traditional MPP architecture data warehouse products that have been launched in the industry very early and have been used today to perform some reports and multi-dimensional offline analysis. Regardless of this process, from the preservation of data to the realization of the value of data, the ability of the entire real-time and unified view of data to support the value of the business will inevitably be lost.

TiDB solution

For TiDB it, our products from initial design to fully consider the technical ability to support enterprise-wide data processing services during , for the manufacture of fresh produce data, TiDB through its ability OLTP Scale, which is highly sensitive for the extension The online data processing capability of the state-of-the-art business ensures that fresh data can be kept fresh after production, and can complete the highly scalable data support service on the online side with extremely high efficiency. At the same time, the most important thing is to ensure its real-time capability.

When companies need to realize effective and rapid value realization of fresh data, TiDB provides Realtime HTAP with real-time data processing and analysis capabilities on the technical side, combined with the ability to mix rows and columns, and transparent computing services, which can provide companies with real-time monitoring. , Real-time large screen, real-time marketing, real-time risk control, real-time query, real-time data warehouse, etc., can not only ensure the unification of data, but also ensure the real-time nature of data consumption, and realize the value of data through multiple dimensions.

TiDB helps digitalize the industry

So how does TiDB provide enterprises with relevant services in all aspects of digital life?

OLTP Scale: For core financial transaction scenarios

Yealink Bank is a licensed Internet bank in China. In addition to traditional banking-related standard services, it mainly has the attributes of the Internet. Many businesses come from Internet traffic. For such users, TiDB has completed the construction of the Yealink Bank's distributed core system project through in-depth cooperation with the users:

• supports the entire core transaction system of

Such as core accounts, core loans, payment systems, user centers, asset securitization, PBOC credit reporting, loan reconciliation, and a series of key transaction services;

• Multi-center multi-active disaster recovery

The multi-center, multi-active disaster recovery in Beijing and Changchun escorts core transactions, and supports the construction of all aspects of the migration from the Oracle system to the TiDB;

• Ready for the future

In the future Internet scenario, providing highly-expandable business support capabilities and the ability to face financial core transaction scenarios in terms of OLTP Scale can help financial institutions complete core-related business support.

OLTP Scale: For financial sensitive trading scenarios

Ping An Life is a well-known insurance company in China. It uses TiDB to complete the online financial online transaction service of "Golden Butler".
This type of business scenario has a very obvious feature: short-term peak transactions .

In the two years from 2019 to 2020, Ping An Life, through the operation of the "1.08 God of Fortune Festival" campaign, has constructed a single-day turnover of more than 100 billion, breaking through the transaction records related to online insurance and wealth management products. Behind this is hundreds of TiDB database instances that provide operational guarantees and complete the entire short-term peak transaction support.

Based on the characteristics of TiDB's high scalability, high throughput, and high online processing capabilities, Ping An Life's Golden Butler project successfully completed the transaction for the entire operating day.

OLTP Scale: Facing financial high-growth transaction scenarios

Japan's top mobile payment company, with the support of the government's policy to promote a cashless society, its payment business is expanding rapidly. Currently, Japan has a population of approximately 100 million, of which 29 million users and 2 million merchants are using the company's payment services, and the recent transaction volume has reached 1 billion.

TiDB through online high throughput capacity and online expansion capacity , successfully helped users to switch from the original Amazon Aurora database to the TiDB platform, and solved the performance scalability problem in the past peak payment transaction process.

OLTP Scale: For retail high-growth transaction scenarios

TiDB has cooperated with the world's leading catering giants to complete its key online high-concurrency and high-expansion core database related to its order center, user center, coupon center, and point center.

At the same time, we will fully study with users how to take advantage of TiDB's cloud-native architecture and combine TiDB and K8S with to perfectly land in the user's production environment. At the same time, TiDB native, highly available, multi-center, and disaster-tolerant guarantee capabilities ensure the support of the entire 7×24 hours business.

Real time HTAP: For financial real-time data service scenarios

I just finished talking about how TiDB uses OLTP Scale, that is, the ability to support highly scalable and sensitive data online to provide enterprises with highly concurrent data processing services. Let's look back at how users can use TiDB to realize the rapid realization of data value.

We are working with the head of the Beijing city commercial banks domestic banks to build a comprehensive real-time data services platform, the system to achieve the following several important functions :

• core system burden of 160a486435366b. The real-time data from the core environment of AS400/DB2 is synchronized and aggregated to TiDB, including mobile banking transactions and other transaction records of over tens of billions of data, providing real-time online query.
• Data Servicing . Plan to aggregate diverse data sources to TiDB to complete the establishment of comprehensive data service capabilities.
• scene diversification . TiDB provides mixed ranks and Real time HTAP capabilities to support more diverse data consumption services.
• Financial-grade security . The Bank of Beijing has implemented various key systems including online payment, credit loans, etc., and has achieved years of "zero failure" operation guarantees.

TiDB has been on the ground in Bank of Beijing for two years. In such a business platform, we will continue to have such zero-fault operation guarantees in the past and continue to provide Bank of Beijing with safe, reliable and stable services.

Real time HTAP: Financial-oriented real-time data service center

In the past year, we have also built a T+0 real-time data service middle station for a financial company. This T+0 real-time data service middle station supports the entire company from real-time detailed data query in the operational domain to real-time Statistical analysis query related data services. At the same time, business support also includes querying transaction data, such as the frequently used QR code payment, back-end acquiring, forwarding, omni-channel services, and its international business.

Because TiDB has online expansion and elasticity capabilities, this T+0 real-time data service middle station uses the online thermal expansion method during the entire expansion process. At the same time, as I said earlier, this example uses TiDB's built-in high-availability technology to complete a financial-grade T+0 data service real-time middle-office service.

Real time HTAP: Shell Financial's data service real-time middle office

Shell Financial is a financial service company under Shell Group, which provides users of Shell with financial leasing financial services in the process of renting and buying and selling houses. We cooperated with the data team of Shell Financial to achieve a set of real-time data service platform. In this system, we successfully integrated TiDB with streaming computing systems such as Flink and Kafka. A set of real-time data processing, data processing and data synchronization data support platform.

Real time HTAP: For logistics real-time data service scenarios

During 2020, we completed a real-time data analysis processing platform with Zhongtong Express, a well-known domestic logistics express company.

Through real-time aggregation of more than ten data message sources of Zhongtong: The ability of TiSpark on top of TiDB is used to aggregate more than 10 Topic data messages, and then they are written into TiDB in real time through TiSpark. Very complicated calculations will be performed on the TiDB side. A wide table with 70+ columns is stored in TiDB: multiple messages are aggregated into Hive dimension table Join and rewritten into TiDB in real time; at the same time, our entire platform also provides very efficient support for Zhongtong Express, Including the support of important business scenarios for secondary dispatch, the entire TiDB platform can process more than 300 million pieces of data in 10 minutes, reaching a QPS of almost 500,000 per second.

In the past six years, through the unremitting efforts of all PingCAP engineers and the community, 160a4864353834 has realized enterprise-oriented OLTP large-scale, high-scalability, and sensitive online transactions in The capacity is . Through the construction of these capabilities, it can help enterprises to complete digital transformation work quickly and efficiently. TiDB has become a key infrastructure for enterprise data preservation and data value realization in the context of digital acceleration.