TDengine 3.0 was released, and the first TDengine Developer Conference ended successfully

On August 13, the cutting-edge data technology exchange event "TDengine Developer Conference" organized by Taosi Data was officially held in Beijing Kuntai Jiarui Cultural Center. At this conference, dozens of distinguished guests, including Tao Jianhui, founder of Taosi Data, Dr. Cui Baoqiu, vice president of Xiaomi Group, and Huang Mingming, founding partner of Mingshi Capital, contributed to the participants interpretation of the future trends of open source and basic software. , as well as data architecture upgrade experience for projects such as Internet of Things, IT operation and maintenance. At the same time, Taosi Data announced the heavy release of TDengine 3.0, and revealed its core features one by one, and many disruptive innovative ideas let the participants take a look at it.

"A hundred schools of thought contend"

Positioning the future of open source and basic software under the collision of ideas

As early as July 2019, TDengine announced the open source of the core code, and announced the open source of the cluster in August 2020. Under the influence of open source power, after 5 years of development, TDengine has nearly 140,000 user instances and developed 100+ Business users.

All along, TDengine's technological innovation is based on user needs. The conference also invited Yan Zheng, the technical director of the IoT product department of JD.com, and Huang Guoshi, senior architect of Zhongtong Technology, to come to the scene to tell the participants about TDengine's use of the JD Cloud IoT industry scenario and the IoT service platform of ZTO's logistics and distribution business. Applied Stories.

As a new business innovation model, the power of open source is obvious to all, but about the value and significance of open source, there are a thousand Hamlets in the eyes of a thousand people, and a million definitions in the hearts of a million people. At the main forum of the conference, Dr. Cui Baoqiu, Vice President of Xiaomi Group, led everyone to return to the essence of open source and explore the meaning of open source.

He said that in the era of the Internet, big data and artificial intelligence, open source is the best platform and model for human technological progress, but open source is not just a method to enhance the brand of technology, or a means of software distribution, its starting point should be Altruism and long-termism. "Virtue is not alone, there must be neighbors." If an open source project adheres to the open source approach of openness, sharing, equality, collaboration, and innovation from the beginning, there will be a lot of co-constructors who will join voluntarily.

To a certain extent, the power of open source has helped TDengine win the domestic and foreign markets and become one of the preferred time series databases for many enterprises. However, if you want to continue to succeed, you must continue to innovate in technology and always be ahead of the "copy". Only Only in this way can the "altruism" and "long-termism" of open source be truly realized.

Tao Jianhui, founder of TDengine

In the sharing of "High-Performance, Cloud-Native Minimalist Time Series Data Processing Platform" brought by Tao Jianhui, the founder of TDengine, we saw the continuous iterative upgrade of TDengine from 1.0 to 2.0 to 3.0, and the development of technology is also feeding back to the open source community , In just three years, the number of stars of TDengine on GitHub has exceeded 18.8k, and the number of issues has reached 15921. The newly released TDengine 3.0 redefines the time series database with features such as cloud native database, minimalist time series data platform, and convenient data analysis.

Although hundreds of domestic alternative products such as TDengine have appeared in China's basic software field, the market is still monopolized by overseas players such as Windows, Linux, MySQL, and Oracle. Whether it is database or operating system software, the domestic market is almost blank. In this situation of development, what is the next step for China's basic software?

Huang Mingming, founding partner of Mingshi Capital, believes that since the rise of China's manufacturing industry in the 21st century, there is an urgent need for matching basic software, and the basic software products rooted in the previous generation of leading manufacturing players have been unable to meet greater output value. , larger scenarios, and the needs of updated players, Chinese companies have a lot to do in the new generation of basic software battlefield. Under this favorable background, the open source of technology can become a means for basic software to occupy the market, which will help enterprises to break the bottleneck of trust and enter overseas markets.

It can be said that by making good use of open source, the domestic software market is expected to stand out. Open source has become an important means to subvert the existing market structure of basic software. But at the moment when open source has become a trend, open source also needs correct "dao" and "skill" to be successful. In the roundtable discussion session of the main forum, several guests had a round of thought collision on this theme.

A common idea in this discussion is that if open source is to succeed, the tonality and innovation of the product itself are very important. The original intention of open source projects must be to solve the common pain points of certain scenarios and do valuable things. In order to continue to attract users and developers, technological innovation is also required to continuously generate fresh blood. In addition, open source projects must have a global mindset, and can open up open source markets at home and abroad by embracing emerging technologies such as cloud native.

"Technology and Innovation"

TDengine 3.0 brings a revolutionary breakthrough in time series database

The release of TDengine 3.0 attracted the attention of many participants. In order to let TDengine community developers, followers and enterprise users have a more comprehensive understanding of 3.0 related technologies and functions, in the core technology session in the afternoon of the conference, the core R&D personnel A series of speeches on the function points and core technologies of 3.0 were conducted.

In recent years, although time-series database products on the market have emerged in an endless stream, many problems in the industry have not been solved, including the high-cardinality problem related to latency, and the complexity brought by relying on third-party tools to implement stream processing, caching, and data subscription functions. There is no real realization of cloud native problems, etc. TDengine 3.0 came into being in this context.

As a true cloud-native time series database, TDengine 3.0 reconstructs the distributed architecture and introduces the RAFT consensus protocol, which can support 1 billion timelines and 100 nodes, completely solving the "high cardinality" problem in time series data processing ; Improve and optimize the support for message queues, streaming computing and caching, and can be used as a minimal time series data processing platform to solve the problem of complex system design and difficult to maintain; The computing engine has been newly designed and optimized to provide Convenient and complete data analysis function.

In addition, from 1.0 to 2.0 to 3.0, the storage engine of TDengine has been upgraded all the way. TDengine 3.0 has carried out a new implementation of the storage engine, including "multi-engine hybrid storage" and "storage optimization for multi-dimensional time series data". Innovation. Previously, TDengine has achieved high storage performance by virtue of the two innovative designs of "one table per device" and "super table". After the optimization and upgrade of the storage engine in 3.0, the storage performance has been improved to a higher level.

In addition to the above-mentioned features of 3.0, TDengine also provides many auxiliary functions: support for more powerful and flexible tag indexing, pre-calculation based on time period, support for Schemaless and more writing protocols, support for Grafana, Google Data Studio and many others A third-party tool that supports incremental data backup, remote disaster recovery, and edge-cloud collaboration.

At present, all the core codes of 3.0 have also been officially released on GitHub, which is convenient for TDengine followers and supporters to download and experience.

"Development and Practice"

Data architecture upgrade experience in IoT and IT operation and maintenance

In scenarios such as the Internet of Things and IT operation and maintenance, massive fragmented devices and massive time series data have brought a series of new requirements and new technical challenges to the development of enterprise platforms. This conference invited a number of corporate customers such as SF Express, Leapfrog Express, OPPO, Yunda and Observation Cloud to share their data architecture upgrade experience in IoT and IT operation and maintenance scenarios.

In the big data monitoring platform of SF Express, the previously adopted OpenTSDB+HBase full monitoring data storage solution has many problems, such as many dependencies, high usage costs, and performance that cannot meet data processing needs. For this reason, SF Technology decided to upgrade the full monitoring data storage solution, and finally selected TDengine among several databases such as IoTDB, Druid, ClickHouse, and TDengine and put it into practical application.

Yin Fei, senior engineer of big data platform R&D of SF Express, said that after the transformation, the SF big data monitoring platform got rid of the dependence on big data components, effectively shortening the data processing link. There are significant improvements in writing and querying. At the same time, the number of physical servers on the server side is reduced from 21 to 3, and the daily storage space required is 93GB (2 copies), which is only about 1/10 of OpenTSDB+HBase under the same copy. , showing great advantages in cost reduction and efficiency increase.

Coincidentally, OPPO's wearable product business has a huge amount of writing and there is a processing requirement for offline/historical data supplementary recording (update). From the perspective of improving user experience, it also needs to have efficient reading and writing efficiency and long consumption of consumer data. time save. The MongoDB/MySQL cluster solution used before has a large back-end storage pressure and requires frequent disk expansion. At the same time, each cluster is relatively independent, and the maintenance and demand development costs are relatively high.

"After going through the road of architecture selection from MySQL to MongoDB to Prometheus and finally to TDengine, there are three main elements for our selection: first, focus on the business, find the key problems that need to be solved in the business, and choose the architecture by comparing different architectures. Second, the system needs to be modified the least, that is, new architectural business capabilities can be built with fewer changes; third, it is the easiest to access, at this point, rich read-write compatible interfaces are very important , which facilitates the expansion of end-side business." said Tang Hengjian, senior back-end R&D engineer at OPPO Cloud Computing Center.

In addition to the above two companies, Yunda, Observation Cloud and Leaping Express also have many problems before the data structure transformation. Yunda previously used MySQL partitioning + indexing to process order scan volume. Faced with the daily data volume of 100 million, MySQL had performance bottlenecks and significantly increased maintenance costs; Observation Cloud previously used InfluxDB, but the HA (high availability) cluster mode could not be leveled. Capacity expansion, the writing performance is about the same as that of a single machine, and the high-availability cluster version of the InfluxDB cloud product on a certain cloud still cannot meet the performance requirements. When large-span time data is used, the performance of the system will drop significantly.

After applying TDengine for system transformation, Yunda's writing speed and query performance have been significantly improved. The writing speed is about 5,000 rows per second, and commonly used queries can basically be completed within 1 second; for the observation cloud, The logical design of TDengine enables it to meet both high reliability and large data-level read and write requirements, and can well support system performance in multi-tenant mode; The 22GB machine is reduced to 1.4GB, the machine resources are greatly reduced, and the operation and maintenance cost is significantly reduced.

From the experience of these enterprises, we can see that for the upgrade of enterprise data architecture, it is the most important to hit the business pain points. In the data architecture transformation practice of many enterprises, TDengine has demonstrated strong read and write performance and data compression capabilities, helping enterprises solve various big data processing problems. It is believed that with the optimization of 3.0, TDengine can be better integrated into the Internet of Things, Internet of Vehicles, IT operation and maintenance and other use scenarios.

Epilogue

The successful holding of this developer conference is not only a landmark event for TDengine to vigorously serve developers and users, but also contributes a lot to the technological progress and future development of the entire industry, and provides many open source developers. Innovative point of view.

The release of TDengine 3.0 completely solves the "High Cardinality" problem in the industry. As a true cloud-native database, it also brings a revolutionary breakthrough to the development of time series databases. The future has come, aiming at the forefront of world science and technology, and the development path of the new data architecture in the digital age has become clear.