Overview of TiDB Containerized Deployment丨Review of the ``Energy Titanium'' Roundtable Forum

Recently, the "Energy Titanium" roundtable forum, which was heavily planned by the TiDB community, came to a successful conclusion. This forum invited Mr. Song Jingchao, the founder of the cloud native community, as the host, and discussed with five senior technical experts from Zhiliu Technology, 360, 58 Group, Autohome, and 160d00ed69e2a8 on the theme of "When the database meets Kubernetes" They have learned their practice and thinking on database containerized deployment and operation and maintenance.

Video review: https://www.bilibili.com/video/BV16q4y1j7UZ

guest introduction:

• presided over : Song Jingchao, Tetrate evangelist, founder of the cloud native community.

• Special guest :

  • Zhang Jintao, Tributary Technology Cloud Native Technology Expert. Contributors to many open source projects such as Kubernetes and Docker, focusing on areas such as Apache APISIX Ingress and Service Mesh.
  • Dai Xiaolei, 360 database operation and maintenance senior expert. 10 years of database operation and maintenance experience, open source enthusiast, TUG MVA / consultant, one of the authors of TiDB in action. Love to summarize and share, and like to challenge new technologies.
  • Liu Chunlei, senior DBA of 58 Group. Responsible for 58 intra-city MySQL, TiDB database, DorisDB operation and maintenance, TUG MOA, content group leader, has extensive database development and operation and maintenance experience, and is keen on database automation and platform construction.
  • Li Xin, DBA of Autohome, is mainly responsible for the database operation and maintenance and development of Autohome.
  • Wang Pengfei, a senior expert, has been engaged in the construction of distributed systems, distributed relational computing, and cloud services for many years. He is currently responsible for the research and development of TiDB Cloud at PingCAP.

Before the roundtable discussion started, Tributary Technology shared with you the development history of K8s.

As we all know, K8s is a component open sourced by Google in 2014 to solve large-scale container orchestration in the production environment. Looking back at the development history of K8s, we can understand that K8s has almost become the de facto standard of cloud native technology around 2017, and is Favored by more and more companies. K8s can become the cornerstone of cloud native and its three major characteristics are closely related. First of all, K8s comes with convenience, including fault self-healing, container orchestration, and service discovery, making applications more convenient. Secondly, the application of K8s in a variety of technical fields such as database, serverless, edge computing, and microservices has expanded the boundaries of its usage scenarios. Third, the stability and security of K8s provide strong support for its production and application. In addition, the continuous improvement of the surrounding ecology has also laid an indelible foundation for the rapid development of K8s.

In recent years, the road to the cloud has attracted much attention. As the infrastructure of enterprise data assets, can the database be combined with cloud native technology, and what kind of sparks will it collide with? This article sorted out the essence of the five guests on the feasibility of database containerization, the process of database migration to K8s, successful cases, and the experience of stepping on pits. I hope that you can refer to and learn from them.

Discussion: Is it feasible to put the database on K8s?

To this question, teacher Dai Xiaolei from 360 gave a positive answer. Although it is feasible to put the database on K8s, the premise is that a certain amount of technical accumulation is required, which is difficult. The database is a stateful service, and the cost of a stateful service on K8s cannot be ignored. In addition, from the perspective of feasibility, some technical problems need to be solved.

Since it is feasible, what are the advantages of database containerized deployment?

believes that there are two advantages of database containerization. One is to reduce and increase efficiency 160d00ed69e5c1. We currently have some clusters that have insufficient resource utilization, waste of CPU or memory, and if a K8s cluster is used, it will These resources are collected as a resource pool, and TiDB is deployed through TiDB Operator, and then the corresponding resources are allocated, which can realize more reasonable resource allocation and centralized management, and save resource costs. second is resource isolation . K8s provides a very good mode of resource isolation. If the cluster management mode of K8s can be used to achieve resource isolation, it is also a good use condition.

In response to these two points, teacher Liu Chunlei of combined with the application of 58 Group's database on K8s raised some daily pain points. In addition to the need for resource isolation, cost reduction and efficiency enhancement, saving machines, improve resource utilization is also one of the advantages of . At present, in all TiDB databases in the same city, TiDB and PD nodes are implemented using virtual machine KVM. TiKV is deployed on a physical machine, which means that multiple instances of a single machine will also compete for resources, which will affect business. If these clusters are all on the cloud, with the same resources, you can deploy twice the number of clusters. This is one of the best experiences. In addition, the resource isolation mode of K8s can reduce the mutual influence of services and bring a more stable experience .

In addition, , teacher Li Xin from also mentioned a rapid deployment problem in many services. K8s can help businesses realize automatic management of instance nodes. From the perspective of development cost , it is also a problem to do a set of database instance cycle management by yourself, and K8s has a set of management procedures. Relatively speaking, you only need to do some K8s extension API development. is very effective for improving efficiency. Help, also help to improve the stability of the database .

Of course, technology selection is rigorous, and we must consider comprehensively when choosing a technical solution. In spite of the above-mentioned advantages, the cloud of databases is bound to be different from offline businesses.

Database is a very large field. In addition to components like TiDB, MongoDB, and ElaticSearch, there is actually an RDBMS. PingCAP's teacher Wang Pengfei believes that it is different from NewSQL or NoSQL in that NewSQL or NoSQL is basically implemented in a distributed manner. Even if a part is broken, it will not affect the use in most cases. A very important feature of K8s is that Pod may hang frequently. During failover, the parts with replicas will be broken and the application will not perceive it. But in MySQL and PostgreSQL, it will be very troublesome if the replicate fails. So when we , it depends on whether the business can accept that the link is broken while running . From the perspective of cost reduction, there are a high probability that several things need to be considered, such as putting together a bunch of pods that are not too busy, and saving machines for use by busier businesses. Then in the process of transferring Pod, it will also involve more frequent application side link breaking. In a similar situation, we need to consider whether the application needs to add a middleware to deal with the above problems.

TiDB Operator architecture diagram

Some middleware of MySQL can solve this problem. In other words, during the migration process of the main server process, it can ensure that the customer is continuously connected and automatically help the customer to complete the retry operation. In this case, there is no line of code at the application layer. At this time, migration may bring greater uncertainty to the choice of technical direction.

In addition, also needs to look at the real cost. In addition to the cost saved by going to the cloud, whether it also increases the research and development cost , so this is a comprehensive cost consideration.

Nothing can be singularly discussed. The advantages of database deployment on K8s are still clear, but the hidden dangers cannot be ignored. Teacher Jin Tao made supplements for the not-so-good points.

• First, potential performance loss , when we put on the database to K8s among the first to face that adds a layer of virtual network, although the network can reduce the impact by underlay way, but inevitably will produce some Performance loss.
• Second, core services is difficult to guarantee . The Pod of K8s may be destroyed at any time, and there is a problem of resource preemption. At this time, the quality of the core service of the database is difficult to guarantee.
• Third, business interaction . The total amount of resources on each machine is fixed. When scheduling, you need to consider the memory problem of the database. When cross-node scheduling occurs, if you want to improve resource utilization, you have to mix it up, which will inevitably cause business interactions. The impact of time.
• Fourth, monitors . Some native features of K8s cannot fully meet the monitoring needs of the business, and often require secondary development to supplement and improve.

The five teachers on the feasibility of deploying the database to K8s and the pros and cons of it have been very detailed. It is still necessary to fully consider various related indicators in the actual application selection in combination with their own business needs.

What do we need to prepare before

basic test

In response to this problem, Teacher Xiaolei believes that the following two aspects need to be considered:

• First of all, for performance testing, use a physical machine or pressure test to see related data, such as how much QPS can reach and what the resource usage looks like. At the same time, we need to pay attention to some monitoring indicators in time to see if the performance can meet the business needs.
• Second, destructive testing. Use some tools of chaos engineering to get rid of MySQL or TiKV Server of TiDB to see what happens and whether the whole failover mechanism can really be pulled up in time. For stateful services, destructive testing is mainly done to get a bottom line and see if the availability of the database is okay.

If the above two aspects of the test are passed, push some edge services to try, and after stabilization, the core business .

However, Mr. Chunlei believes that from the current situation, the performance loss is certain. The more important thing is to pay attention to whether the performance loss affects the use and how to make up for it through other measures. In addition, from a functional point of view, whether the most urgent high-availability switch or daily reading network can normally reach the database, and whether the underlying tools can be used or not also need attention.

Regarding the performance test, there is no doubt that it is a necessary step before the migration, but for the performance test, Mr. Li Xin believes that whether the database is placed on K8s or not, the core goal is to serve the business. Therefore, the main test is to combine the real needs of the business. For example, one is a basic performance test and the other is a model test for the business. It may be that the basic test performance is actually lost, but when it is truly combined with our business model, the performance loss There may be more, this is the part that must be paid attention to in the advance test.

The highly available. Teacher Li Xin believes that is the top priority. How to solve it, how to adapt, and whether the previous operation and maintenance system can be modified after the container is installed, whether the modification can be compatible, and the cost. Consider it.

Architecture and management

Regarding the deployment architecture of K8s, Mr. Li Xin shared the current architecture status of Autohome. Carhome currently has some MySQL deployed on K8s, using self-built K8s clusters, not on the cloud. Basically, a dedicated operation and maintenance team operates and maintains K8s clusters and uses MyQSL Operator to develop them. of. Of course, the development of Operator is a related interface developed by DBA, K8s operation and maintenance team and cloud development team. Management is managed by a dedicated K8s operation and maintenance team. DBA only manages MySQL instances without K8s permissions.

Mr. Jin Tao also has management experience in K8s. He has used self-built clusters and managed clusters. There is also a mode in which node nodes use cloud resources to expand capacity. In this mode, the control plane is self-built, while node nodes are directly expanded through private lines with other cloud vendors.

Each of these methods has advantages and disadvantages.

Self-built K8s cluster

The advantage of self-built clusters is that they are all controlled by themselves, but everyone knows that the purchase cycle of the entire physical machine is relatively long. Correspondingly, it will face the following problems:

• The business is about to engage in major promotion activities, but the physical resources may not be able to be deployed in time.
• If you move some machines from somewhere to the K8s cluster, there are certain requirements for the operating system and its kernel. It may be necessary to go through the installation initialization operation, and then this series of processes is down, not only consumes labor costs, but also time costs .

cloud pull-up dedicated line

Another way is to directly pull a dedicated line on the cloud to achieve capacity expansion at any time, which is very convenient. For example, when the event is over, the resources can be released directly, and the image of the virtual machine can be created on the cloud in advance. The entire virtual machine on the cloud can be pulled up directly with the image, and no additional configuration is required.

Fully managed

The full hosting method is the simplest, but it also faces a problem. The control plane of some cloud vendors is not open. For example, a high-availability control plane with three nodes is set up. Then three nodes will be charged, but these three nodes But it is invisible and cannot be used directly. Of course, in addition to cost, security also has certain risks, after all, the control surface is not in your own hands.

In this regard, Mr. Jing Chao believes that the above architecture is also similar to the hybrid cloud architecture. Regarding how to manage clusters on K8s, Mr. Jing Chao also mentioned some open source software, such as Rancher, KubeSphere, etc. You can also deploy by yourself to manage different Space platform.

extra layer of technology stack, what troubles does it cause at the operation and maintenance level?

daily operation and maintenance need to pay attention to what?

How to be compatible with the existing system, how to transform the existing system, how to do high availability in the container, etc., these issues have been mentioned by Mr. Li Xin. Teacher Chunlei also agreed with this, and besides, he also added some problems that may be encountered, such as:

• Automated tools, automated tools on physical machines need to be targeted for automated development on K8s
• Daily management, is it to pull up another cluster or maintain some daily operations
• Troubleshooting, one more layer and one more consideration, unlike previous physical machines that can be quickly located

In comparison, the container has a certain degree of difficulty . From the perspective of the operation and maintenance personnel, Xiaolei believes that the increase in the technology stack requires learning a new technology for DB, and if K8s is managed by a separate containerized team, the interaction cost will inevitably increase.

In addition, Mr. Jin Tao, who is experienced in K8s operation and maintenance, shared some important concerns and specific suggestions.

• First, at the K8s layer, all components of K8s may have problems. Therefore, when troubleshooting, you first need to pay attention to whether the cluster is healthy. If the cluster is healthy, determine whether the network is healthy, and then specifically on the business side.
• Second, on the business side, you need to check whether the configuration is correct. If the configuration is correct, then confirm whether the specific business is affected.
• Third, the database, each database will have a storage, when the machine hangs, you first need to consider whether the Pod can drift, and if it is a shared storage, you also need to consider performance.

How to ensure the safety and integrity of

In this regard, Teacher Pengfei shared the following points:

• Persistence generally achieved by multiple (>= 3) copies. Integrity can be thought of as ensuring that data is consistent between different participants through transactions.
• security , if you do not consider TiDB's own control of data security, you can think of it as the control mechanism of MySQL.

Here is a list of data security after using K8s, because TiDB supports specifying a key in TiKV. If the key is secure, the data written by TiKV is encrypted. Therefore, the data on the disk is directly read through the K8s mechanism. The data obtained is encrypted data. Normally, there is no way to decode it, so the security is guaranteed.

As for the permissions of the cluster, the TiDB community is in progress. According to Teacher Pengfei, the TiDB community will first deal with the issue of "TiDB can be processed with only one namespace resource permissions" as the first step. This thing is being done and will be directly open sourced to the community after it is done.

Take TiDB as an example, is it feasible to put it on K8s?

on feasibility

In response to this issue, Mr. Peng Fei, the head of TiDB Clould, first shared his views. He believes that TiDB is a distributed database that is naturally adapted to cloud-native deployment .

Cloud native provides a good level of abstraction, which can solve the different deployment environment problems behind this level of abstraction, and provides great flexibility for the operating environment of the business. In addition, cloud native can provide very good flexibility, which can bring a lot of convenience in real operation, you can choose better speed, you can also choose better data durability.

However, TiDB cannot simply be deployed on K8s because TiDB is a distributed system. The most troublesome part of operating and maintaining a distributed system is handling failover. If you want to migrate TiDB to K8s, you need to focus on the following issues:

• First of all, there are many control logics that need to be paid attention to. Currently, these control logics have been implemented in TiDB Operator. TiDB Operator currently has 10w lines of code. Based on this, you can guess how complicated the control logic behind it is. The ultimate goal of TiDB is to use TiDB Operator to automatically solve most of the problems that required operation and maintenance in the past, so as to reduce the user's operation and maintenance costs.
• Secondly, people who write logic are very demanding, and they need to have a very deep understanding of TiDB to handle corner cases. But during production, especially the OLTP system, when a corner case falls into the pit, it may be a P1 accident. Therefore, Mr. Pengfei suggested to put TiDB on K8s as much as possible first, and then use TiDB Operator as much as possible to solve the problem.
• Third, the solution of standardization and automation problems, Mr. Tao Jin thinks, is also very important.

TiDB cluster is deployed to K8s, what should I pay attention to in terms of performance?

• Hard Isolation : Teacher Chun Lei believes that the first is hard isolation from the resources, such as the isolation of CPU, memory, network, disk, etc. If the isolation and comparison are in place, there will not be a particularly large mutual influence. In addition, Xiaolei believes that mutual interference between businesses should be reduced, such as optimization of resource scheduling. Finally, if the K8s cluster can be isolated from the business level, mutual influence can also be avoided.
• Soft isolation : Teacher Pengfei added from the level of soft isolation. He mentioned that hard isolation refers to the use of dedicated machines for dedicated TiDB clusters. In fact, many companies will face operating costs, and there are soft isolation methods for this. Consider, for example, a typical offline and online complementary scenario. The use of soft isolation can also greatly reduce operating costs, which is particularly important for many companies.
• intelligent development : some intelligent development of the back-end can also be considered, for example, you can see the load of the physical machine to understand the approximate fluctuations, so as to make full use of resources, and consider some automated development, such as automated scheduling, etc. , Realize the migration of nodes, and achieve the effect of improving resource utilization.

database containerization is deployed, what is the next step?

Automation and intelligence

Automation and intelligence are two common issues that need to be considered. When in the operation and maintenance and monitoring of multiple clusters, if the robot is split according to the business, then the robot management is a problem. Teacher Xiaolei believes that automation can be considered to solve this problem in this situation. However, automation is also a basic requirement. Teacher Chun Lei believes that platformization and report information display are also a direction, and the other is intelligence, such as automatic fault cure, intelligent expansion and contraction, and operation of databases through automation and intelligence. Dimension lightening.

In this regard, Mr. Li Xin agreed. He believes that the main need to consider how to combine with existing tools to make the entire system more complete, so that users can directly operate the cluster, expand and contract. Especially for the issue of elastic expansion and contraction, the rules for scaling must be rigorous, because once something goes wrong, the consequences will be disastrous.

In addition, some services may be really important. On the physical machine, there is no need to use multiple computer rooms or cross-machine, two places and three centers. However, you need to pay more attention to how to achieve cluster-level high availability on K8s, such as multiple computer rooms. Either multi-cluster deployment, or combined use, to better improve the stability of the database.

Disaster recovery plan of three centers in two places

With regard to the deployment and disaster tolerance of the three centers in the two places, Teacher Peng Fei said that he can explain.

First of all, TiDB cluster inherently supports such deployment from the architecture, but there are also requirements for the infrastructure. If the latency allowed by the business is only 1ms, it means that the time must meet the requirements when calling multiple centers and multiple centers, because this is a complete set. For the OLTP online system, it has higher requirements for the underlying infrastructure.

Then on the cloud, the relevant practice is to deploy TiKV instances in three different AZs. In other words, on the cloud, it can generally be ensured that the TiDB cluster is still available after the entire zone is hung up. However, if it is pushed offline, it is actually applicable to the situation of two places and three centers.

However, the difference between the cloud and the offline is that the communication across the AZ in the same region on the cloud is very stable, and the bandwidth is relatively exaggerated. However, offline deployment does not have such a good infrastructure in all scenarios. Even when the two locations and three centers cross regions, bandwidth and latency will have a relatively large impact. At this time, the deployment mode needs to be fine-tuned. For example, the basic principle of raft is to ensure that the majority is finished, and that the leader falls on two centers in one place as much as possible. Only in the worst case, the election of the leader will go to a single center. These are all issues that need to be considered when deploying offline.

How to better balance performance and safety

Teacher Chun Lei believes that is actually . But safety and performance are inherently mutually or opposite. If you pay more attention to safety, there may be a certain performance loss. If you pay more attention to performance, safety cannot be guaranteed perfectly. But under normal circumstances, there must be a solution in terms of performance.

Teacher Jin Tao believes that in addition to the health status of clusters and pods mentioned above, network latency should also be considered. In addition, in terms of network strategy, Teacher Pengfei believes that TiDB Operator is actually a controller, and its main purpose is to maintain the state of the TiDB cluster through K8s scaffolding. For example, if a TiDB node is missing, TiDB Operator will know it, and use the ability of K8s to make up this node.

So for the TiDB Operator itself, it does not depend on the network. It is used to maintain the state of the TiDB cluster, not the TiDB cluster itself. In fact, in different deployment scenarios, different user scenarios will have different specifications, or specific network plug-ins are selected, these will directly affect the use of the TiDB cluster on the network. In order to adapt to a wide range of situations as much as possible, the only thing that can be done is not to make the deployment of TiDB cluster dependent on specific network plug-ins, but to be able to adapt to all situations in the world.

for novices and sharing experience of stepping on pits

• Pengfei teacher : The biggest suggestion is that K8s is an artifact. Take time to master K8s and be able to move the database to K8s. This will be very exaggerated for the improvement of personal ability, so this matter is hard to return. Difficult, but it must be worth it.
• Jin Tao : First of all, I suggest you have at least a general understanding of K8s. Secondly, we may not be able to decide on the architecture or technology selection of the K8s cluster itself, but we can control the deployment of the application topology. In addition, it is necessary to reasonably set the resource occupation of the application in K8s when using it. Finally, please remember that monitoring must be first.
• Teacher : I am a novice. If a novice is a novice, first input, learn more about relevant knowledge, pay attention to the technology trends of the cloud native community, and try to understand. Then go to toss a test fleet by yourself and get started gradually. These are some of my own learning suggestions for newbies.
• Jingchao teacher : Actually, I think the most important thing to learn is to practice, and then share with you. You can’t be alone and don’t know the changes in the outside world. CNCF gave a specific roadmap for cloud native implementation, starting from containerization, mirroring, CICD, introducing observability (including monitoring, logging, and tracking), etc., slowly moving up to middleware and stateful services. Containerization. Go to the previous step to realize the cloud native landing of the entire company. In this process, everyone will definitely step on the pit, but many problems have been shared with you in the community, so everyone should also participate in the open source community frequently. I believe you can learn a lot from it.