Seeking the present and future of NoSQL, the 5th Techo TVP Developer Summit ended successfully!

introduction

The ever-increasing flood of data and rapidly changing emerging businesses have not only triggered Internet companies to rethink database technology, but also brought about a second myth of database applications in traditional industries. What role does NoSQL, known for its flexibility and easy expansion, play in the process of enterprise digital transformation? Where is its importance? What kind of development will the NoSQL database usher in in the future?

On May 14-15, 2022, the 5th Techo TVP Developer Summit "Counting the Future, More Than Fast - NoSQL Detonates Data Efficiency and Value" officially came to an end. 12 well-known technical leaders from the NoSQL field and Experts discussed the future-oriented NoSQL road with database practitioners from the three directions of performance, efficiency and data value.

1. Day1: NoSQL cutting-edge technology trends

(1) The host opens

The first day of the summit focused on NoSQL's future trend analysis and new technology interpretation, with the chief expert of Tianyi Cloud and Tencent Cloud TVP teacher Hou Shengwen as the host. At the beginning of the meeting, Mr. Hou pointed out that in the era of big data, the continuous and explosive growth of data volume, the flexible and diverse data storage structure, and the increasing changes in emerging businesses, these demands have spawned more and more forms of existence of databases and application systems. The various capabilities of the database provide a lot of scenarios and requirements, and also bring great challenges and requirements to technicians.

(2) Guest speeches

The speech of Lin Xiaobin, Vice President of Tencent Cloud, officially opened the curtain of this Techo TVP Developer Summit. He pointed out that in the wave of digitalization, data has become the most important asset of enterprises, and the database, as an important infrastructure for data storage, has a major task. Over the past decade, infrastructure upgrades, the acceleration of digitalization, and the popularization of cloud computing have brought about a second period of rapid development in the database industry, in which cloud computing service providers have played a huge role.

As cloud database services enter the cloud-native era, the innovative model of database + cloud will further promote the innovation of cloud database technology. NoSQL is a very important branch of the database. It has obvious advantages in massive concurrent access and large-scale data scenarios. At the software architecture level, it has the characteristics of high concurrency, easy expansion, flexibility and ease of use. In short, NoSQL databases have become an indispensable type of database service for modern enterprises.

Tencent Cloud has continued to invest strategically in the NoSQL field, with years of exploration and practice from product design, technological innovation, customer service, and ecological construction. Tencent Cloud public cloud has provided database services including cache, document, time series, KV, etc., and continues to enrich service types. At the same time, Tencent Cloud is also actively promoting external cooperation, such as signing a strategic cooperation agreement with MongoDB.

He finally concluded that Tencent Cloud NoSQL database has covered dozens of typical industry application scenarios including finance, electricity, e-commerce, games, video, etc. In the future, Tencent Cloud will continue to invest in the field of NoSQL database, insist on technological innovation, and serve Enterprises are the foundation, helping enterprises reduce costs and increase efficiency, and improve productivity.

(3) Unleash productivity: Tencent Cloud NoSQL’s follow-up and innovation

Luo Yun, Deputy General Manager of Tencent Cloud Database, gave a keynote speech titled "Unleashing Productivity: Taking Advantage of Tencent Cloud NoSQL to Change and Innovate".

He first introduced the development of NoSQL databases to the attendees over the past ten years. He said that with the rapid growth of Internet business, the requirements of massive users, massive data, real-time experience, and rapid iteration have brought about the rapid development of NoSQL. The development history of Tencent Cloud NoSQL is also following the trend. In the same vein, Tencent Cloud has provided corresponding products and services in a series of scenarios such as cache, KV, document, graph, time series, wide column, and table.

As we all know, Redis has been the most popular NoSQL database product in the past five years, but the application scenarios of Redis have already broken through the scope of caching, especially in storage scenarios, which are more and more widely used, but enterprises use Redis in storage scenarios Faced with the problems of scale, cost, and persistence, the industry has launched Redis persistence products this year, but most products can usually meet 2 to 3 of the 4 demands of the business in terms of performance, cost, persistence, and scale. Rarely can all four demands be satisfied at the same time. Mr. Luo Yun said that Tencent Cloud provides extreme performance and tertiary storage solutions through innovative software and hardware combination to solve performance, persistence, cost, and scale problems. The KeeWiDB team has achieved extreme breakthroughs and achieved:

• Performance (single node): 200,000 reads, 180,000 writes, P99<2ms can be stacked horizontally, and the performance is linearly improved;
• Cost: Tier 3 storage, cold data cost reduced by 97%;
• Persistence: command-level persistence; millisecond-level stable write latency; SSD provides low-cost persistence;
• Large capacity: Single node provides TB-level capacity space; cluster mode provides 100TB capacity space.
  At the end of the sharing, Mr. Luo Yun said that the ultimate goal of technicians is to make the production efficiency of the whole society higher and make the productivity of the whole society better, and hope that these beautiful wishes can be realized by means of technology.

(IV) New hardware empowers innovation: data development trends and software-hardware integration solutions

With the development of technology, the emergence of new hardware brings more possibilities. Intel Database and Big Data Senior Chief Engineer, Tencent Cloud TVP Cheng Congchao & Intel Data Platform Division Optane Product Division China Technology Center Engineering Manager Wu Guoan jointly brought a presentation entitled "New Hardware Empowering Innovation - Data Development Trends and Software Development". "Hardware Convergence Solutions" keynote speech.

Mr. Cheng Congchao introduced to the participants that there are the following very obvious trends in the field of data processing architecture: from Scale up to Scale out; from physical machines to cloud native; from share-everything to share nothing to share everything; in-memory database + Object storage; row storage and column storage coexist. Therefore, end-to-end data processing and end-to-end system optimization have become crucial parts.

Mr. Wu Guoan took over the baton and explained the leading features of Intel Optane's new hardware, which can reduce the cost of data centers, integrate workloads, try to do more with data, and have powerful capabilities in unlocking the potential of data. Intel Optane has outstanding use cases in NoSQL databases, recommendation systems, KV storage, and more.

Wu Guoan finally concluded:

• The development of data analysis requires more integrated solutions of software and hardware;
• Intel® Optane™ brings new memory and storage tiers, making it easier to expand memory and accelerate storage;
• More balanced cost and performance, providing more and better choices for your business;
• New hardware brings new business innovations, making the impossible possible;
• A more open ecosystem, more open and interconnected systems, and more customers embrace new hardware.

(5) Technical insight and practice sharing of MongoDB application data platform

Lin Tao, technical director of MongoDB North Asia, gave a keynote speech entitled "MongoDB Application Data Platform Technical Insights and Practice Sharing".

At the beginning of the sharing, Mr. Lin Tao threw out a shocking data: 70% of the digital transformation of enterprises ended in failure. The reason is that most data infrastructures are still built around traditional relational databases, which cannot meet the needs of current enterprises to use and process data and build programs. Around these needs, choosing to add a dedicated NoSQL data store brings new problems.

Teacher Lin Tao said that changing the complex data storage structure, making development more efficient, making the architecture simpler, and having features such as security and ease of management are exactly what MongoDB wants to achieve. Specifically, MongoDB provides the following product capabilities to help enterprises take off:

• Provide a flexible document model;
• Provide a unified interface;
• distributed architecture;
• Automatic migration of hot and cold data;
• Client field level encryption.
  Finally, Mr. Lin Tao introduced several best practice cases of using MongoDB to build a data platform. Through the data platform, developers and enterprises can accelerate innovation at a lower cost.

(6) Facing the challenge of massive graph data, Tencent's business practice and application in graph database

Xiao Pin, technical director of Tencent Cloud Graph Database, gave a keynote speech titled "Facing the Challenge of Massive Graph Data, Tencent's Business Practice and Application in Graph Database".

Mr. Xiao Pin said that with the explosive growth of big data, the relationship between data has become more complex and diverse, and the correlation calculation and analysis of relational data has become a normal requirement, which has brought about the take-off of graph database technology. Within Tencent, graph technology is widely used in graph databases, graph computing, graph visualization and other fields.

Mr. Xiao Pin focused on key content such as the architecture, performance and use cases of the Tencent graph database KonisGraph. KonisGraph adopts the mode of interface layer, computing layer, distributed cache, and storage layer at the architectural level. The design principle is the separation of storage and computing to ensure the flexibility of deployment. In terms of optimization strategies, optimizations such as asynchronous parallelism, vectorization, batch prefetching, and calculation pushdown are also performed. In addition, it also has unique design concepts in strategies such as caching and indexing.

In addition, Mr. Xiao Pin also shared the design concept and performance effect of GraphIdex graph visualization, Angel Graph graph calculation and its framework performance, and related use cases of KonisGraph graph database. He finally said that KonisGraph will continue to optimize and improve itself in the following directions in the future.

• Fusion graph computing engine;
• GQL syntax support;
• Self-developed storage layer;
• The cache layer is perfect;
• Improve peripheral components and platform capabilities.

(7) Application practice of cloud-native multi-modal NoSQL in feature storage

Zhao Zheng, head of NoSQL development of Tencent PCG Application Architecture Platform Department, brought us a keynote speech entitled "Application Practice of Cloud Native Multi-modal NoSQL in Feature Storage".

Mr. Zhao Zheng introduced that cloud-native multi-modal databases are the current way for many public cloud vendors to provide cloud-hosted storage services. Tencent's self-developed cloud-native multi-model NoSQL database, through abstracting highly scalable data and a highly reusable workflow framework, builds a universal platform base with capabilities such as disaster recovery backup, data tiering, and multiple consistency. It is required to flexibly customize the pluggable storage engine framework and extension API to provide flexible expansion and fast access capabilities of the data model.

On the topic of feature storage, Mr. Zhao Zheng shared the characteristics of "feature" storage, technical challenges, enterprise needs and current solutions and other background information. Based on this background, Tencent Cloud's native multi-mode NoSQL system needs to consider two key points in system design and practice: first, to support multi-level storage capabilities, and second, to separate storage and computing. In terms of storage engine, it is necessary to consider the requirements for both memory and persistent storage. In addition, other requirements can be supported by the multi-modal NoSQL platform. Many technical challenges involved, Mr. Zhao Zheng analyzed to the participants in great detail.

At the end of the sharing, Mr. Zhao Zheng said that Tencent Cloud's native multi-mode NoSQL platform has been connected to five major businesses, with an average daily call volume of more than 100 billion, TB-level full data update less than 1H, and minute-level GB incremental data update, which has significantly reduced the cost. This efficiency-enhancing business benefit. Talking about the future prospect of Tencent Cloud's native multi-mode NoSQL, he painted a wonderful blueprint for the integrated solution of storage and indexing cloud for search and promotion scenarios to the participants.

2. Day2: NoSQL technology practice and application

The second day of the summit focused on the best practice application of NoSQL databases in all walks of life, with Han Feng, the executive director of CCIA and Tencent Cloud TVP, as the moderator.

(1) The host opens

Mr. Han Feng pointed out that with the deepening of digital transformation, more and more attention has been paid to data, and the vigorous development of mobile Internet has generated a large amount of unstructured data such as voice, image, and video. These data contain very rich information. How to unearth the value of these data, how to use NoSQL to help the landing of innovative businesses, and how to enable the rapid development of businesses have become a crucial part of the digital transformation of enterprises.

(2) NoSQL combat in financial scenarios: Redis application practice in WeBank

Hu Panpan, head of WeBank's data platform database and Tencent Cloud TVP, gave a keynote speech entitled "NoSQL Practice in Financial Scenarios: WeBank's Redis Application Practice".

At the beginning of the sharing, Mr. Hu Panpan introduced the pain points of the community version of Redis to the participants. He said that the abnormality of the master node of the sharding architecture has an impact on the cluster, the rights management function is limited, the resource statistics and resource control functions are very limited, and the lack of a unified operation and maintenance and management platform is the pain point encountered by WeBank when using the community version of Redis. .

To this end, WeBank has developed WeRedis, a distributed caching platform based on Redis. In addition to the features of open source Redis, it also has the following features:

• Multi-tenancy and fine-grained authentication;
• resource control;
• Subsystem for resource control;
• High-risk operation isolation;
• Higher scalability;
• higher availability;
• Intelligent analysis and management.
  Hu Panpan said that WeRedis has been applied in all business scenarios of the bank, with 300+ system access, 87 clusters, and 4,000+ instances. Then, he introduced WeRedis's cross-IDC deployment architecture in detail, as well as the pitfalls in performance, high availability, capacity and other issues in the operation and maintenance process, and selflessly shared the WeRedis application development specifications that have been continuously optimized from the review.

Subsequently, he also shared the design and thinking including the WeRedis console function, authentication management and high availability. He finally said that in the future, WeRedis will continue to polish and improve in terms of containerization, upgrading to Redis Cluster 6.0, and persistence architecture.

(3) Tencent Music NoSQL application and practice: building a social business background based on Redis and MongoDB

Li Gewei, the leader of the basic research and development of the national K song, brought a keynote speech entitled "Application and Practice of NoSQL in Tencent Music: Building a Social Business Platform Based on Redis and MongoDB".

At the beginning of the sharing, Mr. Li Gewei introduced the application overview of Tencent Music NoSQL to the participants. According to him, the four major apps of Tencent Music, QQ Music, Kugou Music, Kuwo Music and National K Song, are characterized by multimedia socialization + multimedia content. Extensive use of NoSQL databases.

Teacher Li Gewei said that from the perspective of business architecture, database selection needs to consider three key points: matching the business model, handling massive requests, and balancing costs. Based on this principle, Tencent Music has profound application practices under Redis and MongoDB, two open source database products.

Specific to Redis, the practical dimensions include:

• KV read and write;
• CAS solutions;
• distributed lock;
• light and heavy separation;
• list sorting;
• Bitmap;
• Distributed timer.

Under MongoDB, practice dimensions include:

• Embed or Reference;
• List storage design;
• Replication strategy;
• maxTimeMS caveats.
  Mr. Li Gewei finally introduced that many of the self-developed related technical solutions have been used in Tencent Cloud's database products. He also reminded the participants not to underestimate the database application and operation in the production environment, and to avoid the occurrence of major accidents.

(4) Infinity: WeChat’s new generation of large-scale online storage system

Zheng Jianjun, Leader of Basic Platform of WeChat Technology Architecture Department, gave a keynote speech titled "Infinity: WeChat's New Generation of Large-Scale Online Storage System".

Under the large-scale volume of WeChat, very strict requirements are put forward for high availability, high reliability, performance, cost, and low latency of storage. Teacher Zheng Jianjun first introduced the development process of WeChat storage. The development of WeChat storage can be generally divided into four stages: the first stage of QuorumKV; the second stage is the PaxosStore in 2015; the third stage is the enhancement of the capabilities of PaxosStore during 2018; the fourth stage is a new generation of large-scale online Storage System Infinity. Infinity is a new generation of WeChat's large-scale online storage system, which supports multiple data models, responds to different business scenarios through a unified architecture, and provides linear consistent and near real-time access modes.

Teacher Zheng Jianjun explained that the background of the birth of Infinity stems from the high-performance requirements of the business for the improvement of availability/reliability, more flexible scheduling capabilities, faster expansion speed, recommendation/live broadcast and other business scenarios. Specifically, the core features of Infinity include the following aspects:

• The routing center is completely de-dependent: maintain a high level of non-lease availability to improve scalability;
• Data storage and replication log decoupling: flexibly support storage heterogeneity/plug-in and multiple disaster recovery modes;
• Automatic and extremely fast recovery of machine failures: an order of magnitude improvement in availability/reliability;
• Stable and controllable cluster expansion: Solve the scaling paradox of high-load clusters.
  At the end of the sharing, Mr. Zheng Jianjun also introduced the operational effectiveness of Infinity in WeChat business, including dynamic load balancing, module hybrid scheduling and deployment, and improved utilization. In addition, the faster expansion of the zero-point peak on New Year's Eve, and the case of a large number of heterogeneous models expansion during the Olympic Games, also highlighted the characteristics and capabilities of Infinity.

3. Roundtable Dialogue: The Development and Change of Database Technology in Ten Years

After the wonderful technology sharing session, the roundtable session that inspired thinking and collided with values followed. The topic of this roundtable dialogue is "The Development and Change of Database Technology in the Last Ten Years", hosted by CCIA Executive Director, Tencent Cloud TVP Teacher Han Feng, Intel Database and Big Data Senior Chief Engineer, Tencent Cloud TVP Teacher Cheng Congchao, Tencent Cloud Mr. Luo Yun, deputy general manager of the database, Mr. Hu Panpan, the head of WeBank's data platform database, Tencent Cloud TVP, Mr. Li Gewei, the head of the basic research and development of the national K-song, expressed their opinions and talked, and started a wonderful summit dialogue.

As an old-fashioned technical direction, database technology has gone through decades, and has developed rapidly in the last ten years. The most talked about topic is - what new development trends will database technology have in the next ten years?

Teacher Cheng Congchao's point of view is that under the premise that the underlying computing power has not undergone revolutionary changes, the future development direction is the form of serverless, in-memory, distributed, and computing and storage separation. It is inevitable to consider more security issues, which will bring constraints and new opportunities to the development of database technology.

Mr. Luo Yun said that Tencent has already discussed the development trend of database technology within Tencent, and the internal technical committee has clearly reached consensus on several directions: the first is localization, the second is the integration of multiple models, and the third is the integration of multiple models. It is a breakthrough in the combination of software and hardware under the trend of cloud native.

Mr. Hu Panpan, starting from the business practice of WeBank, proposed the development trend of the hybrid transaction model. He said that the demand for hybrid architecture must exist, and it will become a feature of many databases in the future. Another point of imagination is the emergence of all-in-one products based on emerging open source databases to meet the pain points of small and medium-sized financial institutions that are unable to operate and maintain. In addition, the trend of localization is also a direction recognized by Mr. Hu Panpan.

Mr. Gewei Li, starting from the development of Tencent's internal business, expressed his agreement with the development trend of database technology integration. On the one hand, integration can reduce consumption on the business side, and on the other hand, it can also promote technological innovation and integration. As a fan of open source culture, Mr. Li Gewei said that open source is very likely to be a major development trend of database technology. Through open source collaboration and community nurture, business development will surely be promoted.

Finally, the host, Mr. Han Feng, summarized his understanding and cognition of technology, business, and open source. From the technical side, resource separation, data layering, and serverless applications are all promising directions for database development in the future; on the business side, A large number of differentiated scenarios will lead to the differentiation of databases, but at the same time, there will be a trend of integration after divide and conquer; at the same time, open source is also a good business model, and more resources can be mobilized to do the bottom layer through the open source model. development. Mr. Han Feng said that no matter which direction it starts from, the development of database technology will help enterprises and developers to benefit from it.

Epilogue

The development history of database technology can almost be regarded as the history of technical people constantly breaking through the limits and pursuing higher, faster and stronger struggles. Database technology has been around for decades, with relational databases taking the lead, NoSQL emerging, the strength and stability of commercial database products, and the continuous innovation of open source database products. Now, we have ushered in a wonderful combination of cloud and database. . On this stage where hundreds of flowers are blooming, there is neither a shortage of actors nor a lot of audiences, but there is not too much thinking about technology and judgment of trends.

This Techo TVP Developer Summit, the consistent philosophy is to present the "most informative, interesting and useful" developer summit for developer friends through the most attentive sharing, the most cutting-edge insights, and the most practical viewpoints. Under the wave of transformation in this era, stick to the original intention and original intention of science and technology to be good, and help enterprises and developers to achieve the ideal country of technological development together.