Dry goods are coming! Digital China CIO Shen Yang reveals the TiDB ecology behind Hackathon丨Interview with the judges of TiDB Hackathon

How long have you not written a piece of code just for Have fun?
How long have you not been excited about the realization of a wild idea?
How long have you not been motivated to Coding with your team?
Come to TiDB Hackathon, you can find all these experiences!

TiDB Hackathon 2021 has officially opened registration on December 9, and we have received many team registrations from new and old friends. The theme of this year's Hackathon is "Explore the Sky", which will be held on the 2022.1.8-9 two competition days, with online and offline linkages. The competition invited well-known experts in the database field, community technology experts, and top investor representatives to serve as judges to provide a full range of comments and guidance on the project to escort the experience of the event.

While waiting for the final competition day, the TiDB community conducted a series of interviews with judges and players, asking them to share their experience on open source and Hackathon, and to share their suggestions for Hackathon contestants. This time we invited the special judge of Hackathon-Shen Yang, CIO of Digital China Group, let’s take a look at open source and Hackathon in his eyes~

Introduction to interviewees

Shen Yang, Vice President and CIO of Digital China Group, worked at SAP for 10 years, doing digital transformation for global customers. He joined Digital China in 2017 and is responsible for the digital transformation of Digital China. Participated in the construction of multiple open source projects including TiDB in the past 2 years.

The following is the interview record:

Q1: When was the first exposure to open source?

Shen Yang: I came into contact with open source relatively early, and I have been exposed to open source projects like Linux. The deepest contact is the development platform called ABAP in SAP. Although it is not 100% open source, its code is open to all customers and partners. In fact, it has a little open source flavor.

Because of the openness, all the bugs and problems in the platform can be found and solved by customers or partners by themselves. This reduces the cost of finding and solving bugs very low. Most of the problems in ABAP may not be SAP itself. found. This is my initial deep experience of open source, and it can also be regarded as a community or ecological understanding. This ecology is difficult to break. If it is a completely closed source software, the problem can only be solved by the manufacturer itself, and the solution cost will be very high, and it depends very much on the internal efficiency and process of the manufacturer. A slight problem with a process may have a great impact on customers.

For open source, I myself use three key words to summarize:

• Connection: Use open source to reach more customers, partners, and community participants as quickly as possible;
• Openness: Open source is not only open code, but also requires a more open mind. In the TiDB community, many people often ask "Are you going to make your own version? What impact will it have on TiDB?" In fact, TiDB has always been very open minded;
• Security: In addition to openness and connectivity, security is also very important. Especially in enterprise-level database products, how to ensure security is very important. Because all open source code is open, it is not only open to customers, but also open to potential attackers. For example, the recent Apache Log4j2 security vulnerability has a huge impact. In the future, we need to do better security so that customers can get the best security protection even in an open source code environment.

Q2: What kind of change has taken place in the attitude of Chinese enterprise-level users towards open source?

Shen Yang: Digital China has undergone a pretty big change. We are a very large company, and similar to many companies in China, we will use all kinds of open source components and products. In the past, domestic companies often did not give too much feedback to the community, and even fewer made open source products on their own. Of course, this is also because it is actually very difficult to make a top-level open source project. You may need top-level engineers, but engineers need not have too much pressure to survive in order to calm down and think about long-term things. So for a long time, although open source has been used more in China, the contribution has been relatively small.

In recent years, as domestic Internet companies have enough energy to do open source projects, open source has begun to rise. But what is even more commendable is that in addition to these Internet giants, there are many emerging companies that have also begun to open source, such as TiDB and Apache DolphinScheduler. They are not a team of a large Internet company, but started from a small team and have also made a top open source project. I think the rise of these open source projects has really opened up a new model in China. Open source is not simply a product that can be formed by simply relying on money, team size, and technical strength. It has found a good soil for The open source team can take root even if it is just a small seed.

I think this model has been fully formed in China, and the Chinese market is large enough. As long as your open source product meets some of the needs of Chinese customers, it can be quickly accepted, and it may take a few years to occupy the market. Therefore, I also hope that Chinese companies, even as Party A, can participate in the construction of open source in the future, and even have their own open source projects to form a better Chinese open source ecosystem.

Q3: When did you come into contact with TiDB?

Shen Yang: We have been exposed to TiDB around 2017. I remember that during the "Double Eleven" period, we had a business system related to logistics, and its system data volume and visit volume were many times more than usual. This scenario is very similar to TiDB's elastic scaling scenario, which ensures that the temporary expansion node system does not hang. At that time, there were some hard disk failures that made the system temporarily useless. It would take a lot of time to repair the failure and restore the data.

We finally deployed the TiDB cluster at the end of 2017, and this system has never been suspended until now. Even once, a physical machine of this system went down. We only discovered it after a few days, but it did not affect the business system. This incident gave us a more intuitive understanding of the distributed architecture. Its management cost does not need to be so high. It is not a very sophisticated instrument and requires careful maintenance/maintenance. It is a very practical tool.

Later, we also tried to contribute to the community. For example, participate in TiDB Hackathon, publish a lot of articles about TiDB on Zhihu (TiDB source code analysis), projects in the direction of TiDB for PostgreSQL, etc...

Q4: The original intention of developing the "TiDB for PostgreSQL" project?

Shen Yang: My original background was to do digital transformation in SAP, and the system I faced was different from that of some Internet companies. Most of TiDB's initial customers are Internet companies. The business of Internet companies is actually very flexible. For example, when it encounters tens of millions/hundreds of millions of users at once, the amount of data is greatly expanded in an instant. This type of user pursues flexibility, scalability, and of course the high availability of the system.

However, among enterprise-level customers, there is a very important requirement for various application systems within the enterprise that is complexity. For example, PG's database is better than MySQL in terms of multi-table association and complex queries. In the enterprise-level field, a company is born with many chimneys when it is big. This chimney was not created deliberately by everyone, but because of legal or other factors.

For example, the law stipulates that the financial data of a company cannot be disclosed in advance. Only a few people may know before the financial report. When the financial report is reached, it must be disclosed to everyone. Prior to this, financial data could only be held in the hands of some departments. The same is true for human resources data. It is impossible for everyone to know the company's data. Everything in the company is handled by professional departments. In terms of data, it is naturally composed of many data tables, but its data volume cannot reach the level of the Internet. Its complexity is reflected in the internal authority and the association of multiple tables. PG has done a very good job in this regard. We had some internal applications that we hoped to be able to do on TiDB, but due to the inherent limitations of MySQL, it is difficult to adapt.

We also saw other databases that are more compatible with PG, such as CockroachDB and YugabyteDB, but none of them are 100% compatible with PG. Because PG's grammar and other aspects are very complicated, I believe it is impossible to achieve 100%, so there is room for this field to be done. For a long time, there is not a so-called absolute first place, this time you have hope. Secondly, I think that PG-based distributed databases in China, especially in the open source field, are very few on the market, and there are almost no strong mature competitors.

Therefore, if you can do this in this field, the first aspect is to make up for a shortcoming of TiDB. When there are many customer systems based on PG databases in the future, and you want to migrate to TiDB, you don't need to make a large number of application transformations. There are complex applications and simple applications to apply this thing. It may not be easier to modify a complex application than to modify a database. In the second aspect, we can see that this will form a new direction.

Q5: Any suggestions for other companies that also want to participate in the TiDB community?

Shen Yang: Whether it is a customer or a partner, everyone compares products when they use them. Many articles on the Internet say good things and criticize them. In fact, no product is without shortcomings. I remember that a very early version of TiDB did not even have a pessimistic lock. But this is precisely an opportunity for users or partners. For example, if you are a bank customer yourself, you just started using TiDB and found that there is no pessimistic lock. If you invest in your own team, you may only need to do a little work to lock this pessimistic lock. If done, we can build a little bit more for the community. Even you can sort out a branch of your own and build the whole ecology better. This is a win-win process.

I think in an open source ecosystem, everyone has to think more thoroughly and openly. TiDB is not just a company's product, it can be a collection of many companies' products in the future, and this collection can form a larger product group. It is also possible that a small product or small function in it has nothing to do with any company, and is 100% fully contributed by people in the community. Therefore, I think everyone should actively explore these possibilities. If the fruits that are easy to pick may be picked up late, it will be even more difficult to make contributions later.

Q6: What are your expectations for the TiDB community?

Shen Yang: Regarding the community, I think the TiDB community is already doing very well in China. You can help the whole country and even the world to build open source teams and enterprises. In the future, China will have many, many new open source products, but many people are really just pure engineers thinking at the beginning, they just can write code. Open source is more than just putting the code on GitHub. In terms of community operations, I believe that many engineers simply don't know how to do it.

Our first open source project is a project called DrawDocker. There are only 8 Stars on GitHub, which lacks the power of this kind of community. But another TiDB for PG we made, the TiDB community helped us do a lot of work, and now there are 300 Stars. Many customers and partners in the community are also staring in this direction. They are very interested and have been asking when they can do it. So I very much hope that the capabilities of the TiDB community can be open sourced in the future.

Q7: Will you participate in this Hackathon?

Shen Yang: We are still discussing. If you participate this year, the direction should be similar to last year, or there will be a breakthrough in the direction of TiDB for PG. In fact, we are currently working on a relatively large application internally to adapt our products. We will evaluate whether this project will have some great output recently, evaluate our workload, and see if we will participate in this year's Hackathon.

Q8: What are your expectations for this year's Hackathon? What advice do you have for the contestants?

Shen Yang: For this Hackathon, I very much look forward to some players who have innovative breakthroughs. I hope that everyone in Hackathon will not only solve some small problems, but also hope that the solved problems can incubate more creative products in the future, such as products like Chaos Mesh. It may be just a concept or a small tool at the beginning, but later it not only made a great contribution to TiDB, but also can be used in all other cloud-native fields.

I hope that this time Hackathon can be a top-level activity driven by interest and goal in the open source industry in China. I also hope that this year will be able to produce some projects that have some contributions to the future software industry, to the future open source industry, in terms of infrastructure, or some breakthroughs. Although it may still be difficult to make a product in the short term, even if it is just a demo or a simple demonstration in a short time, I hope that everyone's ideas will try to think in a larger direction.

TiDB Hackathon 2021

The theme of this year's Hackathon is "Explore the Sky", which will be developed during the 2022.1.8-9 two competition days, with online and offline linkage. From the Ti planet to the firmament of the universe, we use Hacking to connect to a wider range of ecology, giving the contestants imagination space to fully develop. In this edition of Hackathon, we have fully upgraded the track and opened up four major tracks: kernel, tools, ecology, and "∞". Participants, whether they are database developers, database upstream and downstream ecology, or database users, can find suitable ones. In the field, explore unlimited possibilities around TiDB.

TiDB Hackathon 2021 will officially start registration on December 9. Contestants can form a team of 1-4 participants and complete their work within the competition period. The winning team will receive bonuses, technical and resource support. The competition invited well-known experts in the database field, community technology experts, and top investor representatives to serve as judges to provide comprehensive comments and guidance on the project, eliminate knowledge blind spots, and escort the experience of the event. In addition, there are top investors who sit here all the way, so that your strength can be seen by more people.

For more details: Explore the Sky丨Come to TiDB Hackathon 2021 to explore the infinite possibilities