TiDB Operator + Amazon Web Service, exploring the best practices of cloud native databases

Author introduction: Tianyi, TiDB Community Architect. He has worked in Fidelity Investment and Softbank Investment, has rich experience in database high availability scheme design, and has in-depth research on the high availability architecture and database ecology of TiDB, Oracle, PostgreSQL, MySQL and other databases.

Nowadays, the topic of database going to the cloud has attracted more and more attention. As an important basic software, what kind of changes will the database face in the cloud native era? When the database meets cloud native, what spark will it collide with? Recently, on Amazon Cloud Developer Meetup, TiDB Community Architect Wang Tianyi shared TiDB and cloud native practical development experience.

This article interprets from the following three aspects: TiDB Operator and cloud native :

• What is a cloud native database;
• Why cloud native database TiDB should embrace Kubernetes;
• Best practices of TiDB on AWS.

What is a cloud native database

what is cloud native

Any technological change must be thought first. I think cloud native is a methodology for running applications on the cloud.

We can split cloud native into two parts: cloud and native. The so-called cloud must mean that the application is located in the cloud instead of in the traditional data center. For example, the files in the cloud disk are in the cloud instead of being stored on the local hard disk. For the native interpretation, I think it is born into the cloud, and the application needs to consider the cloud environment from the beginning of the design, and run in the best posture on the cloud.

All in all, cloud native is fully utilizes the elasticity + distributed advantages of the cloud platform. is a technology that is born on the cloud, grows on the cloud, and is used on the cloud.

Cloud native features

Since the concept of cloud native was born in 2013, its definition has been continuously improved. Cloud native has the following four characteristics:

• Continuous delivery: The concept of agile development was introduced about ten years ago. The purpose of agile development is to respond to changes in user needs, to achieve frequent releases and rapid delivery. In some grayscale releases and canary release scenarios, several different versions may provide services at the same time.
• Containerization: Containerization makes our development, testing, and production environments highly unified. In the research phase, we can also use commands such as docker compose to quickly build a research environment.
• Microservice: Microservice is an independently published application service. Applications communicate through rest API and can be deployed, updated, expanded, and restarted independently.
• DevOps: DevOps emphasizes the efficient coordination of the relationship between development and operation and maintenance. Quickly deploy applications to the production environment through automated deployment and CI tools.

The essence of cloud native is to take advantage of technology dividends such as cloud computing resource pooling and platform scale to create more business value.

What is cloud native database

Cloud native database is a database service built, deployed and distributed through a cloud platform. It is distributed in the form of PaaS and is often called DBaaS. Compared with traditional databases, cloud native databases provide better accessibility and scalability.

cloud native database

• The cloud native database must have an automatic fault-tolerant mechanism to achieve automatic migration of downtime and automatic fault isolation to ensure high availability of applications. This is the most basic feature of cloud native databases.
• Cloud native databases must have good elastic scaling capabilities, which can automatically scale according to CPU load, memory usage, etc., and expand in seconds.
• For elastic scaling, flexible billing methods are required, which can be paid by traffic, paid by resources, or combined package services, with multiple pricing strategies to achieve the goal of reducing costs and increasing efficiency for users.
• The cloud-native database should be easy to manage, provide good alarm monitoring services, and simplify operation and maintenance operations, and transform from self-service operation and maintenance to automatic operation and maintenance.
• Cloud native databases should also have a good security isolation mechanism. It includes both multi-tenant resource isolation and network security isolation.

All of these features are for the ultimate user experience. low learning costs, low operation and maintenance costs, and low price costs. uses cloud native databases.

say that TiDB is a cloud native database

We first look at TiDB basic architecture :

• TiDB Server: Responsible for accepting client connections, executing SQL analysis and optimization, and finally generating a distributed execution plan.
• TiKV Server and TiFlash Server: used to store data, TiKV is a row-based storage engine, and TiFlash is a column-based storage engine.
• PD: The entire TiDB stores the source information of the cluster and is responsible for the scheduling operations of the cluster. It is the brain of the cluster.

storage node :

• TiKV: Provides a key-value row storage engine for distributed transactions.
• TiFlash: Columnar storage engine, mainly for acceleration of analysis type scenes.

TiDB Cluster itself has high availability and high scalability, and it fits well with the concept of cloud native.

Why embrace Kubernetes

Kubernetes history

When it comes to cloud native, there are two topics that cannot be avoided, one is containerization and the other is container orchestration. In what way should TiDB go to the cloud? I have to mention Kubernetes here. From the perspective of the development history of Kubernetes, Docker has solved the problem of containerization, and Kubernetes has solved the problem of container orchestration:

• In 2013, the Docker project was released, making the sandbox technology of the entire operating system at your fingertips;
• In 2014, the Kubernetes project was released, and the container design model was formally established;
• In 2017, the implementation standards for the Kubernetes project were established;
• In 2018, Kubernetes has become a cloud-native operating system.

A Preliminary Study of Database × Kubernetes

I once designed such an architecture in my last company: the underlying database uses the three main structure of MariaDB, Galera, and Cluster. MariaDB, Galera, and Cluster are master-master replications composed of a stateless instance. The upper layer uses stateless ProxySQL for SQL routing, and then to a different MariaDB. The middle layer uses ZK for service registration and discovery, and controls the upper ProxySQL and the lower MariaDB. Then such a structure is naturally suitable for deployment on Kubernetes.

At first, we were worried about Kubernetes on the DataBase, so we put the relatively lightweight Proxy node in Kubernetes, and the DataBase layer was still managed on the physical machine. In order to deal with accidents, in addition to the proxy deployed on Kubernetes, proxy nodes are also deployed on the physical machine. So even if the Kubernetes cluster fails completely, we can easily deal with this situation.

After accumulating more than half a year of Kubernetes maintenance experience, we decided to put a set of more marginalized clusters on Kubernetes. Since then, both the proxy on Kubernetes and the Database on Kubernetes have been completed.

What attitude should

Is it appropriate to put stateless TiDB nodes on Kubernetes? In fact, this is consistent with the idea of putting proxy on Kubernetes. If there are plans to use Kubernetes on TiDB, then maintaining a TiDB + Kubernetes cluster like this should be the best way to deal with unfamiliar environments, especially for disaster tolerance learning and network configuration, which are very helpful. Many companies used TiDB in this way before going to Kubernetes.

Students who know TiDB should know that the bottom layer of PD is based on ETCD. Is there any good way to use Kubernetes for ETCD? Because ETCD is a stateful service, it may not be suitable for naked deployment on Kubernetes. We can learn from the well-known etcd-operator in the industry to develop a set of pd-operator.

It may be more complicated for TiKV. As a database product, Kubernetes must be very cautious on the storage layer. Before going to Kubernetes, we can take a look at how TiKV is implemented.

We split the data in the storage layer TiKV into small data blocks, which are called Regions in TiKV.

In order to ensure correctness and availability, each region corresponds to a raft group, and each region has at least three copies through raft log replication. It can be seen that at least at the TiKV layer and is a stateful service. When a TiKV node hangs, we can't simply use the PV of the faulty node to create a new pod .

We all know that Kubernetes judges node failures based on whether the Kubelet service can report the node status normally. Imagine that if Kubelet cannot be started normally, but the container in the node is still operating normally, and then pv is linked to other nodes, there will be a double-write problem.

Therefore, it is imperative to use Operator to manage the state of TiDB.

TiDB best practices on AWS

Why choose TiDB Operator

What is TiDB Operator? Simply put, TiDB Operator is a combination of CRD + Controller. The CRD is responsible for declarative management, and the Controller is responsible for driving the display state of the TiDB cluster to the desired state. For example, our definition of a TiDB cluster is 3 TiDB and 3 TiKV. For some reason, a TiDB node is down. Then the actual state is 2 TiDB nodes, and the expected state is 3 TiDB nodes. At this time, the controller can start a new TiDB node.

CRD defines many types of TiDB Cluster components and ecological components. For example, the TiDB Cluster type is defined for the TiDB cluster, the TiDB Monitor type is defined for the monitoring, and the DM Cluster type is defined for the DM synchronization component.

Next, let's take a look at how TiDB-Operator automatically maintains the TiDB cluster.

TiDB-Operator application practice

During deployment, TiDB Operator will select the best Kubernetes native object for each component. PD is essentially a component of ETCD. TiDB is a stateless service. You don't need to care about what deployment is most appropriate. TiDB Operator has hidden this information for us. PD is developed based on ETCD and needs to do peer discovery, and Operator will break up TiKV containers and automatically add store-labels, set which availability zone and which computer room each container comes from, and assist PD in realizing the highly available topology of the region. With TiDB Operator, we can copy and spread the original factory's operation and maintenance experience through code. on Kubernetes by simply passing a statement in yaml format. Of course, just creating is not enough, Operator also provides best practices for operation and maintenance.

When upgrading, we send an upgrade request to TiKV. At this time, the TiKV node needs to be shut down and then replaced with the specified version of the image. Because TiKV is a stateful service, we need to do some operations before shutting down the TiKV node. For example, call the PD interface to expel the region leader from the current TiKV node. At this time, TiKV will not accept read and write requests. After that, you can successfully rebuild the TiKV container and upgrade TiKV to the specified version. Of course, after the upgrade is over, we have to call the PD interface to move the region leader back, so as to reciprocate and roll the upgrade.

This is a working three-copy TiKV node. According to the raft protocol, more than half of the nodes need to survive to provide services to the outside world, that is, it can tolerate at most one node failure.

When the service of TiKV 1 is abnormal, combining PD store information and Kubernetes container information, through some probe methods, we can know whether TiKV is abnormal.

So should failover be performed when a failure is detected? This is not the case. Failover too fast may cause frequent switching due to the jitter of network resources or CPU resources. Too slow failover may reduce the high availability and throughput of the cluster.

At this time, TiDB Operator will perform automatic failover. Under what circumstances and how often failovers are required, these are all left to the TiDB Operator for judgment, and how to perform failovers are also handed over to TiDB Operator for execution.

Of course, a lot of opposition has circulated in the industry. Whether we should deploy Database on Kubernetes or not, we still need to "think twice":

• thinks back . We must not only consider how to move the data on the physical machine to Kubernetes, but also think about the way out. When the Kubernete operation and maintenance operation is too complicated, or temporarily for some other reasons we cannot cover too much technology stack, how to withdraw from the Kubernetes platform Next, Binlog and TiCDC tools can help us.
• think about the danger . With the addition of a layer of Kubernetes and Operator technology stack, the risk point becomes higher. That means we have to do more testing. The Kubernetes master node is down, and we can still maintain external services. Of course, the master node itself also supports Keepalived + Haproxy high availability. The Node node is down, and we can move the Pod on the node to other Node nodes. Kubernetes is all down, and there are PV files saved locally. Even if a catastrophic failure causes damage to the PV file, we still have regular backups for recovery.
• think change . Kubernetes itself is a platform born for change. On Kubernetes, upgrade operations, expansion and contraction, have become easier.

In addition, please think about two more questions:

• Would you like to believe that Kubernetes is the future?
• you ready?

If you have not prepared the relevant technology stack yet, or you want to feel the charm of TiDB on the Kubernetes platform in advance, and feel the charm of cloud native, we provide you with a DBasS service based on AWS-TiDB Cloud. With TiDB Cloud, users can not only get reliable database services, but also enjoy professional operation and maintenance services, while avoiding high learning costs.

On TiDB Cloud user can freely choose TiDB nodes for computing and for storage TiKV & TiFlash node can real demand by the amount of use, according to demand payment amount, will play to the advantage of native cloud Extreme, cost reduction and efficiency increase .

In 2014, Gartner used the term HTAP in a report to describe a new type of application framework to break the gap between OLTP and OLAP. It can be applied to both transactional database scenarios and analysis. Database scenario.

HTAP enables real-time business decisions. This architecture has obvious advantages: it not only avoids cumbersome and expensive ETL operations, but also allows faster analysis of the latest data. This ability to quickly analyze data will become one of the core competitiveness of future enterprises.

Some users may need to do some query-type analysis by operators at regular intervals. At this time, we can analyze the last TiFlash node of the previous day, and synchronize the data from TiKV to the TiFlash node for the operation staff to analyze on the next day. Therefore, the characteristics of elastic scaling on the cloud and the HTAP scenario are inherently highly matched.

The above is about the practical experience related to TiDB Operator and Amazon Web Service cloud native, I hope it can be helpful to everyone.