Technology upgrade & industry upgrade, TiDB + Yiche creates a super car carnival

Lead : The annual double eleven is here again and again, the best gift for technical people is big promotion technical guide ! After these years of development, the big promotion has long been not only limited to the e-commerce industry. Now all walks of life actually adopt similar methods for operating activities. There are 818 in the automotive industry, 618 in e-commerce, 11.11, etc., etc. The big promotion scenario poses many new challenges to basic software including databases, and also accumulates many best practices.

Before Double Eleven, PingCAP will conduct a series of in-depth discussions with users such as Autohome, Bitauto, JD.com, and Zhongtong, hoping to reveal what kind of technical problems are hidden behind the soaring sales volume year by year? What technical architecture can be used to stably carry the traffic peak?

The "Big Promotion" Carnival in the Automobile Industry

Established in 2000, Bitauto is one of the earliest auto Internet platform companies in China. It provides professional and rich Internet information services for auto users, and enhances users' experience in the process of car selection, car purchase, car use and car exchange. .

During this year's "818" period, Yiche and Zhejiang Satellite TV jointly launched a car industry "Spring Festival Gala" that integrates car craft shows, celebrity song and dance performances and celebrity variety Super 818 Car Carnival Night ". While bringing audiovisual feasts and car purchase benefits to car users, the gala also introduced over 150 half-price cars with value-for-money benefits. Spectators can watch the gala while grabbing good cars sold at a 50% discount, as well as car purchase red envelopes and deduction coupons. Multiple discounts, such as direct reduction in car prices, will provide you with real car purchase benefits. As of the end of the party, 224 million people watched the live broadcast on the entire platform, 43,900 online orders were obtained, and the cumulative turnover (GMV) was 6.42 billion yuan.

The big promotion debut of Bitcar

In Bitche’s 818 carnival, there are many application scenarios for databases, among which real-time data Kanban is one of the main application services. Kanban can display real-time data performance of the special topics, activities, traffic, clues, and interactions of the Bitauto 818 car shopping festival, and is the overall data output of the big data platform.

Since this car carnival night of Bitauto is an interactive live event on Taiwan and the Internet, Shake (red envelopes, half-price cars, Bitcar coins) and voting for live broadcast programs in the main venue and sub-venues are the ones with the highest user participation and the largest data traffic. Link. During the entire activity, not only the database is required to be able to store massive amounts of data, but it is also required to be able to cope with scenarios such as high concurrency and low latency. The database here will not only be used as a data storage medium, but also as a data source for real-time calculations, and cooperate with traffic data to achieve real-time broadcast of second-level data.

Database and Flink are two very important components in the entire system. Flink's data sources include database and business traffic data. Therefore, the database must not only meet the second-level real-time push of data, but also support Flink's highly concurrent read and write requests.

Tian Zhen, the person in charge of Bitauto database, said frankly that Bitauto is the first big promotion this year. There is not much experience and the quantity is not easy to predict. Many needs are only put forward at the end. To be on the safe side, the DBA team made a downgrade plan when designing the big promotion plan, but no one wants to actually implement the downgrade, which is too unfriendly to the user experience. Therefore, the entire DBA team will focus on stress testing, and plan the database stress testing program according to the usual two orders of magnitude (100 times).

In the beginning, the preferred database considered by Bitcar was still MySQL. However, in the process of stress testing, in order to ensure the real-time performance of the calculation results, real-time tasks will frequently write large quantities of data to the database. The MySQL master-slave delay is high. In extreme cases, the MySQL master-slave switchover will take too long. The database is temporarily unavailable. At the same time, real-time tasks will continue to write large amounts of data, causing the disk to become full. In the live broadcast process where every second counts, this is definitely intolerable.

In the desperate situation, Tian Zhen thought of TiDB.

"Swimming in the swimming middle school" TiDB is in danger

In fact, Tian Zhen had been in contact with TiDB a long time ago. At that time, he thought that TiDB was an overseas product, and his understanding of TiDB mainly started from overseas communities. However, for reasons of caution, Tian Zhen hopes to thoroughly research the product before officially launching it. This big promotion gave the two sides a perfect opportunity for cooperation. He described the process as "swimming in a swimming middle school".

The technical support of the TiDB community has given the Bitauto DBAs a very important help. From the official establishment of the project in July, completed the selection, scheme design, pressure test, and online deployment of in less than one month. 818" carried the big promotion flow smoothly and without danger.

818 Car Spree Data Kanban Business Architecture Diagram

Throughout the 818 event, TiDB was used as the core database for the 818 car carnival data board. Bitauto has prepared two sets of TiDB clusters, which correspond one-to-one with the real-time computing main and backup solutions. Business R&D writes data into two clusters at the same time by double writing. The query of one part of the business is connected to cluster 1, and the query of the other part of the business is connected to cluster 2. When a problem occurs in one of the clusters, the application side will switch to the other cluster. Both TiDB clusters are deployed with 3 TiDB Servers, 3 PD Servers, 6 TiKV nodes, and 2 TiFlash nodes. In addition, 4 machines have been prepared for expansion to prevent the data volume from skyrocketing and the cluster cannot support it.

In the end, the data volume during the Bitauto 818 car carnival reached more than 10 times the usual amount. When Cai Xukun appeared at the end of the live broadcast, the database traffic directly quadrupled, and he almost launched a one-click expansion plan that was prepared in advance. In the whole process, SQL 999 of 818 car carnival data kanban business was always controlled within 8ms, SQL 99 was around 3ms, and QPS reached 62k.

Red envelope shake business architecture diagram

At the same time, TiDB is also used as a disaster recovery solution in the red envelope shake business to avoid the situation that MySQL is unavailable due to the skyrocketing business traffic. Once unavailability occurs, the business party can directly switch the database to TiDB. TiDB needs three roles as a data source, real-time calculation dimension table, and real-time calculation result storage engine in the entire business . TiDB pushes data to Kafka in real time through TiCDC. In order to ensure that TiCDC is stable and efficient, BitCar creates a TiCDC task for each library in TiDB, and pushes the data to the designated Kafka in real time, and then Flink is responsible for sending the data in the same TOPIC The data belonging to different database tables are parsed and diverted to the TOPIC corresponding to the database tables for use in real-time computing services. Real-time computing tasks consume TiDB data in Kafka for business logic calculations. At the same time, it also needs to query the corresponding dimensional data from TiDB, and finally output the calculation results to TiDB.

The challenge of rapid growth: unifying the technology stack

The extreme scenarios of big promotions can always find some problems that are not usually noticed. In the rapid development of Bitauto, many businesses have introduced a lot of technology stacks, such as Lambda, Kappa and other big data architectures, in order to quickly iterate and go online. Real-time data warehouses such as Kylin, Druid, Clickhouse, etc. But there are only 6 people in the DBA team of Bitauto. It is undoubtedly a big challenge to manage so many technology stacks.

The unified technology stack has become the best choice for the Bitauto DBA team. Taking advantage of the opportunity of this big promotion, Bitauto hopes to replace Kylin, Druid, and Clickhouse with TiDB, simplifying the technology stack, and the DBA team can also return to full-time work. superior.

TiDB's HTAP architecture is a fusion architecture that mixes transactional transactions and analytical processing. Because they are all in the same architecture and the same set of data, it solves the problem of data flow delay in the real-time data warehouse of Bitauto. Data does not need to be copied from the OLTP database, and enters the analysis tool after a long ETL cleaning process.

And TiDB is perfectly compatible with MySQL , which means that DBAs and developers do not need to make any changes, as long as they can use SQL. In the past, when Hadoop or Spark was used, the relatively high cost of learning caused certain barriers to use.

After this battle, Bitauto's business side has added a lot of expectations and trust to TiDB. In the future, Bitche’s advertising, media platform, website, placement data, and advertising effects are all expected to be seen in real time. Tian Zhen hopes to borrow TiDB to cover the entire hybrid technology stack of Bitche and connect with other data streams. All of these require TiDB HTAP. Support real-time data warehouse.

For enterprises, addition to supporting business innovation, is also a large-scale training and full-link exercise of its own technical architecture . Through the extreme test of the big promotion, the company's IT structure, organizational process, and talent skills have all been greatly improved. The experience and thinking in the big promotion will also accelerate the company's daily business innovation rhythm, improve technology-driven innovation efficiency, and create a new growth engine.