Hello everyone, my name is Z!
The previous article realized the installation and deployment of MySQL database on the K8s cluster based on KubeSphere deployment. The deployment method adopts the form of graphical interface. This article will introduce how to use GitOps to deploy MySQL. All YAML files involved in the deployment process will be version-managed using Git and stored in the Git repository. Therefore, this article also touches on the basic operations of GitOps.
Native K8s deploys MySQL using GitOps
In the last article, we completed the deployment of single-instance MySQL through KubeSphere, so how to operate native K8s? What is GitOps and how can it be implemented?
What is GitOps (excerpt from the web)
- GitOps is a set of practices for managing infrastructure and application configuration using Git, which refers to an open source version control system.
- GitOps operates with Git as a single source of truth for declarative infrastructure and applications.
- GitOps uses Git pull requests to automatically manage the provisioning and deployment of infrastructure.
- The Git repository contains the entire state of the system, so traces of changes to the system state can be viewed and audited.
- GitOps is often used as an operation and maintenance model for K8s and cloud-native application development, and enables continuous deployment of K8s.
- GitOps is a way of continuous delivery. Its core idea is to store the declarative infrastructure and applications of the application system in the Git repository.
Prepare a resource allocation list - sorting out ideas
We know that the necessary skill to play K8s is to write a resource configuration list, and generally use a YAML format file to create our expected resource configuration.
At this time, we also have to write the MySQL resource configuration list by hand? I'm panicking, I can't remember all the parameters.
NO! NO! NO! The moment of opportunism has come, and the key to the previous sale has been opened here.
Previously, we have created the resource configuration of MySQL through the graphical interface of KubeSphere, and a great feature of KubeSphere is that you can directly edit the YAML file of the resource online.
When we create resources, we can directly edit the YAML file to create resources. You can also modify existing resources by editing YAML.
Of course, you don't need a graphical interface, you can directly use the command line at the bottom of K8s to obtain the output in YAML format, and then edit it.
Sort out the resources included in the resource configuration list involved in MySQL.
- StatefulSet (stateful replica set)
Service
- Inside the cluster (Headless)
- Outside the cluster (custom service)
- ConfigMap
- Secret
Next, we will obtain these resource configuration lists separately.
Prepare a Resource Configuration Checklist
ConfigMap
Configuration -> Configuration Dictionary , find mysql-cnf , click the three vertical dots on the right, and click Edit YAML .
Open the edit YAML page, you can directly copy all the content, or you can click the download icon in the upper right corner to download the file (you can also use the upload icon to upload the file).
The obtained configuration of the existing network cannot be used completely. It needs to be modified to clear some metadata information automatically added by the system.
mysql-cfm.yaml of the current network.
kind: ConfigMap
apiVersion: v1
metadata:
name: mysql-cnf
namespace: lstack
annotations:
kubesphere.io/creator: lstack
data:
custom.cnf: |-
[mysqld]
#performance setttings
lock_wait_timeout = 3600
open_files_limit = 65535
back_log = 1024
max_connections = 512
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
thread_stack = 512K
sort_buffer_size = 4M
join_buffer_size = 4M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 64M
thread_cache_size = 768
interactive_timeout = 600
wait_timeout = 600
tmp_table_size = 32M
max_heap_table_size = 32MModified mysql-cfm.yaml .
kind: ConfigMap
apiVersion: v1
metadata:
name: mysql-cnf
namespace: lstack
data:
custom.cnf: |-
[mysqld]
#performance setttings
lock_wait_timeout = 3600
open_files_limit = 65535
back_log = 1024
max_connections = 512
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
thread_stack = 512K
sort_buffer_size = 4M
join_buffer_size = 4M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 64M
thread_cache_size = 768
interactive_timeout = 600
wait_timeout = 600
tmp_table_size = 32M
max_heap_table_size = 32MSecret
Configuration- > Secret Dictionary , find mysql-secret , click the three vertical dots on the right, and click Edit YAML .
mysql-secret.yaml of the current network.
kind: Secret
apiVersion: v1
metadata:
name: mysql-secret
namespace: lstack
annotations:
kubesphere.io/creator: lstack
data:
MYSQL_ROOT_PASSWORD: UEA4OHcwcmQ=
type: OpaqueModified mysql-secret.yaml .
kind: Secret
apiVersion: v1
metadata:
name: mysql-secret
namespace: lstack
data:
MYSQL_ROOT_PASSWORD: UEA4OHcwcmQ=
type: OpaqueHere to say, the value in Secret is encrypted by base64, so the MYSQL_ROOT_PASSWORD here should be encrypted by base64 with the actual password.
base64 decryption.
[root@ks-k8s-master-0 ~]# echo "UEA4OHcwcmQ=" | base64 -d P@88w0rdbase encryption.
[root@ks-k8s-master-0 ~]# echo -n "P@88w0rd" | base64 UEA4OHcwcmQ=
StatefulSet
Application Load -> Workload -> Stateful Replica Set , find mysql , click the three vertical dots on the right, and click Edit YAML .
mysql-sts.yaml of the current network.
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: mysql
namespace: lstack
labels:
app: mysql
annotations:
kubesphere.io/creator: lstack
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
creationTimestamp: null
labels:
app: mysql
annotations:
logging.kubesphere.io/logsidecar-config: '{}'
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
- name: volume-rca2zx
configMap:
name: mysql-cnf
items:
- key: custom.cnf
path: custom.cnf
defaultMode: 420
containers:
- name: lstack-mysql
image: 'mysql:5.7.38'
ports:
- name: tcp-mysql
containerPort: 3306
protocol: TCP
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_ROOT_PASSWORD
resources:
limits:
cpu: '2'
memory: 4000Mi
requests:
cpu: 500m
memory: 500Mi
volumeMounts:
- name: host-time
mountPath: /etc/localtime
- name: data
mountPath: /var/lib/mysql
- name: volume-rca2zx
readOnly: true
mountPath: /etc/mysql/conf.d/custom.cnf
subPath: custom.cnf
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: default
serviceAccount: default
securityContext: {}
schedulerName: default-scheduler
volumeClaimTemplates:
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: data
namespace: lstack
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: glusterfs
volumeMode: Filesystem
status:
phase: Pending
serviceName: mysql-1dpr
podManagementPolicy: OrderedReady
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 0
revisionHistoryLimit: 10Modified mysql-sts.yaml .
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: mysql
namespace: lstack
labels:
app: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
volumes:
- name: host-time
hostPath:
path: /etc/localtime
type: ''
- name: volume-cnf
configMap:
name: mysql-cnf
items:
- key: custom.cnf
path: custom.cnf
defaultMode: 420
containers:
- name: lstack-mysql
image: 'mysql:5.7.38'
ports:
- name: tcp-mysql
containerPort: 3306
protocol: TCP
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_ROOT_PASSWORD
resources:
limits:
cpu: '2'
memory: 4000Mi
requests:
cpu: 500m
memory: 500Mi
volumeMounts:
- name: host-time
mountPath: /etc/localtime
- name: data
mountPath: /var/lib/mysql
- name: volume-cnf
mountPath: /etc/mysql/conf.d/custom.cnf
subPath: custom.cnf
volumeClaimTemplates:
- metadata:
name: data
namespace: lstack
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: glusterfs
serviceName: mysql-headlessService
First create the Headless service, apply load- > service- >, find mysql-xxxx(mysql) , click the three vertical dots on the right, and click Edit YAML .
mysql-headless.yaml of the current network.
kind: Service
apiVersion: v1
metadata:
name: mysql-1dpr
namespace: lstack
labels:
app: mysql
annotations:
kubesphere.io/alias-name: mysql
kubesphere.io/creator: lstack
kubesphere.io/serviceType: statefulservice
spec:
ports:
- name: tcp-mysql
protocol: TCP
port: 3306
targetPort: 3306
selector:
app: mysql
clusterIP: None
clusterIPs:
- None
type: ClusterIP
sessionAffinity: None
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStackModified mysql-headless.yaml .
kind: Service
apiVersion: v1
metadata:
name: mysql-headless
namespace: lstack
labels:
app: mysql
spec:
ports:
- name: tcp-mysql
protocol: TCP
port: 3306
targetPort: 3306
selector:
app: mysql
clusterIP: None
type: ClusterIPLook at the custom mysql-external service, application load -> service ->, find mysql-external , click the three vertical dots on the right, and click Edit YAML .
mysql-external.yaml of the current network.
kind: Service
apiVersion: v1
metadata:
name: mysql-external
namespace: lstack
labels:
app: mysql-external
annotations:
kubesphere.io/creator: lstack
spec:
ports:
- name: tcp-mysql-external
protocol: TCP
port: 3306
targetPort: 3306
nodePort: 32529
selector:
app: mysql
clusterIP: 10.233.36.71
clusterIPs:
- 10.233.36.71
type: NodePort
sessionAffinity: None
externalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStackHere is a point to explain the nodePort parameter. If the K8s cluster is controllable, it is recommended to plan a set of service port usage specifications. Each service that requires nodePort specifies a fixed port, which is conducive to the standardization of operation and maintenance.
The modified mysql-external.yaml (note that the nodePort parameter is not specified).
kind: Service
apiVersion: v1
metadata:
name: mysql-external
namespace: lstack
labels:
app: mysql-external
spec:
ports:
- name: tcp-mysql-external
protocol: TCP
port: 3306
targetPort: 3306
selector:
app: mysql
type: NodePortCommit the MySQL resource configuration manifest to the Git repository.
Through the above operations, we have obtained the resource configuration list of MySQL.
I have obsessive-compulsive disorder and like to store in categories, so I used 4 files, mysql-headless.yaml and mysql-sts.yaml are combined in one file. Of course, you can also put them in a configuration file.
- mysql-external.yaml
- mysql-sts.yaml
- mysql-secret.yaml
- mysql-cfm.yaml
Submit the resource configuration manifest to the Git repository
Choose GitHub as the main repository and Gitee as the sync repository (manual).
All k8s resource configuration manifest files in this series of documents use a public repository. It is recommended that each service create a configuration repository in the production environment, which is conducive to more refined version control.
In this article, in order to demonstrate the use of the main and standby warehouses, two Git services, Github and Gitee, are selected. In actual use, it is recommended to choose Gitee for a better user experience.
Create a new warehouse in GitHub, the warehouse name is k8s-yaml , add a README file to initialize the warehouse, and click Create repository to confirm the creation.
Clone the code repository back to your local.
$ git clone git@github.com:devops/k8s-yaml.git
$ ls k8s-yaml
README.mdCreate a new folder, edit the MySQL resource configuration list with your favorite text editor (vscode is recommended), and put the files into the newly created folder.
For future extensibility, a second-level directory named single is created here to store the resource configuration manifest file of the single instance.
$ mkdir -p k8s-yaml/mysql/single
$ ls -l k8s-yaml/mysql/single
total 32
-rw-r--r-- 1 z staff 646 5 11 19:23 mysql-cfm.yaml
-rw-r--r-- 1 z staff 266 5 11 19:31 mysql-external.yaml
-rw-r--r-- 1 z staff 134 5 11 19:23 mysql-secret.yaml
-rw-r--r-- 1 z staff 1911 5 11 19:31 mysql-sts.yamlSubmit the edited list of resource configuration files to GitHub.
$ cd k8s-yaml
$ git add .
$ git commit -am '添加MySQL single资源配置清单'
$ git pushCheck it out on GitHub to see if the code is committed.
Next, synchronize the resource configuration list to the Gitee backup repository.
- This article adopts the method of manual push synchronization (personal habit)
- Gitee also supports automatic synchronization of GitHub repositories (more convenient)
Create a new warehouse in Gitee, the warehouse name is k8s-yaml , the type is private by default, and click Create .
After the creation is complete, you can go to the repository settings and modify it to open source.
After the creation is completed, because we did not choose to initialize the configuration of the repository when we created it, a help page will be displayed by default, telling you how to submit the code to the repository.
Because we already have a code repository, we choose the configuration method of the existing repository and submit the existing code to Gitee.
Operate according to the help prompts, pay attention to the origin we have to replace with gitee .
$ git remote add gitee https://gitee.com/zdevops/k8s-yaml.git
$ git push -u giteeCheck on Gitee to see if the code is committed.
Modify the Gitee repository to be open source (optional).
Gitee Warehouse-> Management- > Warehouse Settings- > Basic Information , whether to open source at the end, select Open Source , Warehouse Public Notice , check all three, and click Save .
After modification, your code repository is open source and visible to everyone.
GitOps first experience - deploy MySQL on K8s cluster
The MySQL resource configuration list has been stored in the Git online repository, and now we will start our GitOps experience journey.
Log in to the master node of k8s and perform the following operation tasks.
It is recommended to create an independent operation and maintenance management node for the production environment to manage the entire cluster. You can refer to "Playing with k8s based on KubeSphere - Notes on Creating Operation and Maintenance Management Nodes"
Install Git.
$ yum install git -yCreate the devops directory, I choose /opt directory as the root directory of devops.
$ mkdir /opt/devops
$ cd /opt/devops/Download the code for the k8s-yaml repository from Gitee.
$ git clone https://gitee.com/zdevops/k8s-yaml.git
$ ls k8s-yaml/
mysql README.mdSince it is the same test environment, first clean up the existing MySQL service.
$ kubectl get secrets -n lstack
NAME TYPE DATA AGE
default-token-x2gzv kubernetes.io/service-account-token 3 31d
mysql-secret Opaque 1 2d20h
$ kubectl get configmaps -n lstack
NAME DATA AGE
kube-root-ca.crt 1 31d
mysql-cnf 1 47h
$ kubectl get service -n lstack
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
glusterfs-dynamic-afe88cf4-86b1-4215-833a-534c5f779a22 ClusterIP 10.233.13.188 <none> 1/TCP 2d
mysql-1dpr ClusterIP None <none> 3306/TCP 2d
mysql-external NodePort 10.233.36.71 <none> 3306:32529/TCP 47h
$ kubectl get statefulsets -n lstack
NAME READY AGE
mysql 1/1 2d
# 清理
$ kubectl delete statefulsets mysql -n lstack
statefulset.apps "mysql" deleted
$ kubectl delete service mysql-external -n lstack
service "mysql-external" deleted
$ kubectl delete service mysql-1dpr -n lstack
service "mysql-1dpr" deleted
$ kubectl delete secrets mysql-secret -n lstack
secret "mysql-secret" deleted
$ kubectl delete configmaps mysql-cnf -n lstack
configmap "mysql-cnf" deletedOne-click deployment of MySQL using the resource configuration checklist.
$ cd /opt/devops/k8s-yaml/
$ ls
mysql README.md
$ kubectl apply -f mysql/single/Verify the results and find that the StatefulSet is not created, analyze the problem.
$ kubectl get statefulsets -n lstack
No resources found in lstack namespace.
# 一开始我以为我遗漏了配置文件，ls看一眼，发现文件都在
$ ls
mysql README.md
$ cd mysql/
$ ls
single
$ cd single/
$ ls
mysql-cfm.yaml mysql-external.yaml mysql-secret.yaml mysql-sts.yaml
# 确认一下文件内容，发现文件也有内容
$ vi mysql-sts.yaml
# 再次执行，发现了端倪，为啥只有service/mysql-headless 的资源配置，没有statefulset
$ kubectl apply -f mysql-sts.yaml
service/mysql-headless unchanged
# 再次确认，发现编辑文件的时候遗漏了一点，当一个配置文件有多种资源定义时，不同资源的配置直接需要用"---"分隔。修改配置文件再次执行，发现执行成功。
$ vi mysql-sts.yaml
$ cd ..
$ kubectl apply -f single/
$ kubectl get statefulsets -n lstack -o wide
NAME READY AGE CONTAINERS IMAGES
mysql 1/1 31s lstack-mysql mysql:5.7.38
$ kubectl get pods -n lstack -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mysql-0 1/1 Running 0 35s 10.233.116.59 ks-k8s-master-2 <none> <none>Going back to our KubeSphere management console, we found that the mysql workload can also be displayed in the interface, which also verifies that the operations on the native k8s will also be directly reflected in the KubeSphere management console.
Second experience of GitOps
Just taking advantage of the above problems, I experienced GitOps for the second time. We directly modified mysql-sts.yaml on the deployment server, and the modified result was verified successfully.
To demonstrate more scenarios of GitOps, make changes directly on the deployment server, and then submit to the online code repository.
In actual work, I modify it on my office computer, submit it to the online code repository, and then deploy the server to pull the updated code.
The modified mysql-sts.yaml , due to space problems, only the key parts are shown here. For the complete configuration of StatefulSet, see the Gitee repository or the previous article.
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: mysql
namespace: lstack
labels:
app: mysql
...
---
kind: Service
apiVersion: v1
metadata:
name: mysql-headless
namespace: lstack
labels:
app: mysql
spec:
ports:
- name: tcp-mysql
protocol: TCP
port: 3306
targetPort: 3306
selector:
app: mysql
clusterIP: None
type: ClusterIPSubmit the modified code to the code repository.
# 修改后查看git仓库的变化
$ git diff
diff --git a/mysql/single/mysql-sts.yaml b/mysql/single/mysql-sts.yaml
index f775920..1eded9c 100644
--- a/mysql/single/mysql-sts.yaml
+++ b/mysql/single/mysql-sts.yaml
@@ -1,3 +1,4 @@
+---
kind: StatefulSet
apiVersion: apps/v1
metadata:
@@ -68,6 +69,7 @@ spec:
storageClassName: glusterfs
serviceName: mysql-headless
+---
kind: Service
apiVersion: v1
metadata:
# 本地提交代码变更
$ git commit -am '修复mysql statefulset配置不生效问题'
# push到在线代码仓库，有一个warning可以忽略，也可以按提示执行
$ git pushCheck the Gitee online repository for changes.
On the personal office computer, synchronize the updated code.
# 更新代码
$ git pull
# 同步更新后的代码到Github
$ git push -u originCheck the GitHub online repository for changes.
Experience GitOps again
Simulate a business scenario and experience GitOps again.
- After MySQL goes online, due to the increase in business volume, the max_connections in the initial configuration parameter is too small and needs to be increased.
- After the configuration parameter adjustment is completed, update the online configuration and restart the service (do not restart the production environment database easily, this requirement can be solved by temporary modification).
- This is just to simulate a simple example to show you GitOps. All configuration files in actual use are recommended to use Git for version control.
Edit the mysql-cfm.yaml file in the MySQL resource configuration list of the local Git repository, and modify the max_connections from 512 to 1024.
Commit changes to the Git online repository.
# 提交本地修改
$ git commit -am '修改mysql-cnf中max_connections的值'
# 提交到Github
$ git push
# 同步到Gitee
$ git push -u giteeLog in to the operation and maintenance management node, update the Git code, and run it again.
$ git pull
$ kubectl apply -f mysql/single/
# 查看ConfigMap的变化
$ kubectl get configmaps mysql-cnf -n lstack -o yaml
apiVersion: v1
data:
custom.cnf: |-
[mysqld]
#performance setttings
lock_wait_timeout = 3600
open_files_limit = 65535
back_log = 1024
max_connections = 1024
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
thread_stack = 512K
sort_buffer_size = 4M
join_buffer_size = 4M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 64M
thread_cache_size = 768
interactive_timeout = 600
wait_timeout = 600
tmp_table_size = 32M
max_heap_table_size = 32M
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"custom.cnf":"[mysqld]\n#performance setttings\nlock_wait_timeout = 3600\nopen_files_limit = 65535\nback_log = 1024\nmax_connections = 1024\nmax_connect_errors = 1000000\ntable_open_cache = 1024\ntable_definition_cache = 1024\nthread_stack = 512K\nsort_buffer_size = 4M\njoin_buffer_size = 4M\nread_buffer_size = 8M\nread_rnd_buffer_size = 4M\nbulk_insert_buffer_size = 64M\nthread_cache_size = 768\ninteractive_timeout = 600\nwait_timeout = 600\ntmp_table_size = 32M\nmax_heap_table_size = 32M"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"mysql-cnf","namespace":"lstack"}}
creationTimestamp: "2022-05-12T07:20:07Z"
name: mysql-cnf
namespace: lstack
resourceVersion: "8928391"
uid: 1b7322cf-f11e-445d-a2ba-b42a90ade469
# 重启mysql pod使配置生效
$ kubectl delete -f mysql/single/mysql-sts.yaml
$ kubectl apply -f mysql/single/mysql-sts.yaml
# 查看mysql容器内部配置是否更新
$ kubectl exec mysql-0 -n lstack -- cat /etc/mysql/conf.d/custom.cnf
[mysqld]
#performance setttings
lock_wait_timeout = 3600
open_files_limit = 65535
back_log = 1024
max_connections = 1024
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
thread_stack = 512K
sort_buffer_size = 4M
join_buffer_size = 4M
read_buffer_size = 8M
read_rnd_buffer_size = 4M
bulk_insert_buffer_size = 64M
thread_cache_size = 768
interactive_timeout = 600
wait_timeout = 600
tmp_table_size = 32MRemember! The above example is just to let everyone experience GitOps. Do not restart the database server in the production environment unless you know what you are doing.
Now it has been verified that our MySQL configuration is available and relatively stable. We record this good state to avoid future changes and damage, and the original correct configuration cannot be found again.
Tag the current Git code on our personal computer and record the current status (it can also be operated through the management interface of the online repository).
# 打tag -a tag名字 -m tag描述
$ git tag -a v0.1 -m 'mysql version v0.1'
# 查看现有tag
$ git tag -l
v0.1
# 查看tag详细信息
$ git show v0.1
tag v0.1
Tagger: devops <devops@163.com>
Date: Thu May 12 18:15:34 2022 +0800
mysql version v0.1
commit 180f97ac96da504a0b46eb4871ef423f64fde093 (HEAD -> main, tag: v0.1, origin/main, origin/HEAD, gitee/main)
Author: devops <devops@163.com>
Date: Thu May 12 17:48:18 2022 +0800
修改mysql-cnf中max_connections的值
diff --git a/mysql/single/mysql-cfm.yaml b/mysql/single/mysql-cfm.yaml
index e24d96d..50d1778 100644
--- a/mysql/single/mysql-cfm.yaml
+++ b/mysql/single/mysql-cfm.yaml
@@ -10,7 +10,7 @@ data:
lock_wait_timeout = 3600
open_files_limit = 65535
back_log = 1024
- max_connections = 512
+ max_connections = 1024
max_connect_errors = 1000000
table_open_cache = 1024
table_definition_cache = 1024
# 将tag推送到远程服务器
$ git push -u origin --tags
$ git push -u gitee --tags
# 线上服务器验证（图略）The operation and maintenance management server updates the code and switches to the specified tag ( note! When using Git, you must develop the habit of git pull before each operation ).
## 更新代码
$ git pull
## 切换到v0.1
$ git checkout -b v0.1Through the above few waves of operations, we can see that all our configuration changes are managed by Git, which completely records the full life cycle management of the configuration. By branching or tagging the warehouse, it is convenient for us to switch to any existing configuration. record status.
High-availability deployment of MySQL (reserved to occupy the pit)
There is no requirement for high-availability deployment for the time being, so the deployment of MySQL in high-availability mode is not involved, but there are some thoughts to keep in mind.
current practice
- Don't make trouble for yourself, buy the RDS of the cloud service provider directly if you have high availability requirements.
- You really need to build it yourself and deploy the master-slave outside the K8s cluster.
possible future directions
- MySQL master-slave deployment on K8s
- Operator
- Helm
Remaining problem
This part of the content is also a necessary skill for MySQL operation and maintenance. I have no experience and cannot share some content, and some content will be introduced in the <<K8s production practice road based on KubeSphere>> series of documents.
- MySQL database backup
- MySQL high availability deployment
- MySQL security hardening
- MySQL tuning
MySQL Performance (Benchmark) Testing
Operation and maintenance must be aware of their own operation and maintenance environment. Before MySQL goes online, performance (benchmark testing) must be performed, which helps to understand the ideal state that our database server can achieve. This introduction is just the skin, just to tell you some basic introductory knowledge. For more details and more in-depth content, please refer to other more professional documents.
Performance (Benchmark) Test Tool Installation
Tool selection (sysbench)
- Cloud vendors use this tool to demonstrate the performance of their own database products
- It is said that many DBAs also like to use
sysbench tool installation
- installation tool
# 导入软件源
$ curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash
# 安装sysbench
$ yum install sysbench -y- Verify - execute the command to view the version
$ sysbench --version
sysbench 1.0.20Performance (Benchmark) Test
Test program
- Test parameters
index value Threads 8/16/32 Amount of data in a single table 100000 Number of tables 16 Performance
index illustrate TPS Transactions Per Second , that is, the number of transactions executed by the database per second, subject to the number of successful commits. QPS Queries Per Second, that is, the number of SQL executed by the database per second (including insert, select, update, delete, etc.). RT Response Time , the response time. Including average response time, minimum response time, maximum response time, and query ratio of each response time. The comparison to focus on is the top 95-99% maximum response time. Because it determines the short board in most cases. Concurrency Threads Concurrency, the number of query requests that can be processed per second.
Prepare test data
Using the database we created on k8s involves database operation commands and requires a terminal to log in to the container to run.
Create a test database sbtest in advance, and give root permission to remotely manage all databases from any IP.
Don't do this in the production environment, you must follow the principle of minimization!
# bash
root@mysql-0:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.38 MySQL Community Server (GPL)
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create database sbtest;
Query OK, 1 row affected (0.02 sec)
mysql> grant all privileges on *.* to 'root'@'%' identified by 'P@88w0rd' with grant option;
Query OK, 0 rows affected, 1 warning (0.02 sec)- Test whether the database can be connected
# 安装mysql客户端，下面的示例是在k8s节点上安装的，由于系统是最小化安装，所有会安装很多依赖。实际测试可以起一个mysql的pod或是用其他的mysql客户端工具。
$ yum install mysql -y
# 测试MySQL服务连通性 -h 是k8s节点的IP -P 是mysql外部服务的端口号
$ mysql -h 192.168.9.91 -P 32529 -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.38 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]>- Prepare test data
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root --mysql-password=P@88w0rd --mysql-db=sbtest --table-size=100000 --tables=16 --threads=8 --events=999999999 --report-interval=10 --time=100 /usr/share/sysbench/oltp_common.lua prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Initializing worker threads...
Creating table 'sbtest6'...
Creating table 'sbtest2'...
Creating table 'sbtest8'...
Creating table 'sbtest3'...
Creating table 'sbtest7'...
Creating table 'sbtest5'...
Creating table 'sbtest1'...
Creating table 'sbtest4'...
Inserting 100000 records into 'sbtest3'
Inserting 100000 records into 'sbtest6'
Inserting 100000 records into 'sbtest1'
Inserting 100000 records into 'sbtest4'
Inserting 100000 records into 'sbtest7'
Inserting 100000 records into 'sbtest5'
Inserting 100000 records into 'sbtest2'
Inserting 100000 records into 'sbtest8'
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest11'...
Inserting 100000 records into 'sbtest11'
Creating a secondary index on 'sbtest5'...
Creating a secondary index on 'sbtest1'...
Creating a secondary index on 'sbtest6'...
Creating a secondary index on 'sbtest4'...
Creating a secondary index on 'sbtest7'...
Creating a secondary index on 'sbtest2'...
Creating a secondary index on 'sbtest8'...
Creating table 'sbtest13'...
Inserting 100000 records into 'sbtest13'
Creating table 'sbtest9'...
Inserting 100000 records into 'sbtest9'
Creating table 'sbtest14'...
Creating table 'sbtest12'...
Inserting 100000 records into 'sbtest14'
Inserting 100000 records into 'sbtest12'
Creating table 'sbtest15'...
Inserting 100000 records into 'sbtest15'
Creating table 'sbtest16'...
Creating table 'sbtest10'...
Inserting 100000 records into 'sbtest16'
Inserting 100000 records into 'sbtest10'
Creating a secondary index on 'sbtest11'...
Creating a secondary index on 'sbtest13'...
Creating a secondary index on 'sbtest9'...
Creating a secondary index on 'sbtest12'...
Creating a secondary index on 'sbtest14'...
Creating a secondary index on 'sbtest15'...
Creating a secondary index on 'sbtest10'...
Creating a secondary index on 'sbtest16'...- Execute Test - 8 Thread Test
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root --mysql-password=P@88w0rd --mysql-db=sbtest --table-size=100000 --tables=16 --threads=8 --events=999999999 --report-interval=10 --time=100 /usr/share/sysbench/oltp_read_write.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Running the test with following options:
Number of threads: 8
Report intermediate results every 10 second(s)
Initializing random number generator from current time
Initializing worker threads...
Threads started!
[ 10s ] thds: 8 tps: 88.46 qps: 1782.38 (r/w/o: 1249.19/355.46/177.73) lat (ms,95%): 267.41 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 8 tps: 84.31 qps: 1678.47 (r/w/o: 1173.42/336.43/168.62) lat (ms,95%): 277.21 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 8 tps: 70.20 qps: 1413.82 (r/w/o: 990.21/283.20/140.40) lat (ms,95%): 369.77 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 8 tps: 47.30 qps: 946.00 (r/w/o: 662.20/189.20/94.60) lat (ms,95%): 484.44 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 8 tps: 43.80 qps: 875.99 (r/w/o: 613.19/175.20/87.60) lat (ms,95%): 484.44 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 8 tps: 60.70 qps: 1213.08 (r/w/o: 849.69/242.00/121.40) lat (ms,95%): 411.96 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 8 tps: 53.90 qps: 1078.22 (r/w/o: 754.42/216.00/107.80) lat (ms,95%): 376.49 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 8 tps: 56.49 qps: 1127.98 (r/w/o: 790.11/224.88/112.99) lat (ms,95%): 397.39 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 8 tps: 50.60 qps: 1014.59 (r/w/o: 709.56/203.82/101.21) lat (ms,95%): 434.83 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: 8 tps: 54.70 qps: 1093.12 (r/w/o: 765.22/218.50/109.40) lat (ms,95%): 390.30 err/s: 0.00 reconn/s: 0.00
SQL statistics:
queries performed:
read: 85582
write: 24452
other: 12226
total: 122260
transactions: 6113 (61.10 per sec.)
queries: 122260 (1221.96 per sec.)
ignored errors: 0 (0.00 per sec.)
reconnects: 0 (0.00 per sec.)
General statistics:
total time: 100.0494s
total number of events: 6113
Latency (ms):
min: 35.63
avg: 130.89
max: 951.86
95th percentile: 390.30
sum: 800129.59
Threads fairness:
events (avg/stddev): 764.1250/4.14
execution time (avg/stddev): 100.0162/0.01Execute Test - 16 Thread Test
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root --mysql-password=P@88w0rd --mysql-db=sbtest --table-size=100000 --tables=16 --threads=16 --events=999999999 --report-interval=10 --time=100 /usr/share/sysbench/oltp_read_write.lua run sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2) Running the test with following options: Number of threads: 16 Report intermediate results every 10 second(s) Initializing random number generator from current time Initializing worker threads... Threads started! [ 10s ] thds: 16 tps: 114.41 qps: 2310.22 (r/w/o: 1621.18/458.63/230.41) lat (ms,95%): 369.77 err/s: 0.00 reconn/s: 0.00 [ 20s ] thds: 16 tps: 106.35 qps: 2111.86 (r/w/o: 1474.74/424.41/212.71) lat (ms,95%): 383.33 err/s: 0.00 reconn/s: 0.00 [ 30s ] thds: 16 tps: 80.40 qps: 1612.01 (r/w/o: 1129.21/322.00/160.80) lat (ms,95%): 623.33 err/s: 0.00 reconn/s: 0.00 [ 40s ] thds: 16 tps: 63.40 qps: 1266.80 (r/w/o: 886.80/253.20/126.80) lat (ms,95%): 539.71 err/s: 0.00 reconn/s: 0.00 [ 50s ] thds: 16 tps: 57.20 qps: 1145.91 (r/w/o: 802.74/228.78/114.39) lat (ms,95%): 549.52 err/s: 0.00 reconn/s: 0.00 [ 60s ] thds: 16 tps: 69.91 qps: 1408.31 (r/w/o: 987.57/280.92/139.81) lat (ms,95%): 511.33 err/s: 0.00 reconn/s: 0.00 [ 70s ] thds: 16 tps: 78.00 qps: 1547.22 (r/w/o: 1080.51/310.70/156.00) lat (ms,95%): 484.44 err/s: 0.00 reconn/s: 0.00 [ 80s ] thds: 16 tps: 79.50 qps: 1599.87 (r/w/o: 1122.58/318.29/159.00) lat (ms,95%): 520.62 err/s: 0.00 reconn/s: 0.00 [ 90s ] thds: 16 tps: 67.80 qps: 1354.83 (r/w/o: 947.62/271.61/135.60) lat (ms,95%): 539.71 err/s: 0.00 reconn/s: 0.00 [ 100s ] thds: 16 tps: 73.90 qps: 1474.10 (r/w/o: 1030.80/295.50/147.80) lat (ms,95%): 502.20 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 110950 write: 31700 other: 15850 total: 158500 transactions: 7925 (79.00 per sec.) queries: 158500 (1580.05 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 100.3103s total number of events: 7925 Latency (ms): min: 41.24 avg: 202.44 max: 1198.81 95th percentile: 511.33 sum: 1604328.52 Threads fairness: events (avg/stddev): 495.3125/4.03 execution time (avg/stddev): 100.2705/0.03- Execute Test - 32 Thread Test
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root --mysql-password=P@88w0rd --mysql-db=sbtest --table-size=100000 --tables=16 --threads=32 --events=999999999 --report-interval=10 --time=100 /usr/share/sysbench/oltp_read_write.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Running the test with following options:
Number of threads: 32
Report intermediate results every 10 second(s)
Initializing random number generator from current time
Initializing worker threads...
Threads started!
[ 10s ] thds: 32 tps: 140.10 qps: 2825.04 (r/w/o: 1981.25/560.39/283.39) lat (ms,95%): 450.77 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: 32 tps: 124.41 qps: 2515.49 (r/w/o: 1763.43/503.24/248.82) lat (ms,95%): 549.52 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: 32 tps: 95.90 qps: 1887.10 (r/w/o: 1316.70/378.60/191.80) lat (ms,95%): 733.00 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: 32 tps: 81.80 qps: 1656.59 (r/w/o: 1164.89/328.10/163.60) lat (ms,95%): 707.07 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: 32 tps: 82.60 qps: 1638.41 (r/w/o: 1143.51/329.70/165.20) lat (ms,95%): 657.93 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: 32 tps: 94.34 qps: 1905.84 (r/w/o: 1336.62/380.65/188.58) lat (ms,95%): 623.33 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: 32 tps: 87.86 qps: 1739.86 (r/w/o: 1215.31/348.73/175.82) lat (ms,95%): 634.66 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: 32 tps: 84.40 qps: 1705.48 (r/w/o: 1196.49/340.20/168.80) lat (ms,95%): 759.88 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: 32 tps: 80.50 qps: 1580.71 (r/w/o: 1101.70/318.00/161.00) lat (ms,95%): 612.21 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: 32 tps: 81.40 qps: 1661.90 (r/w/o: 1167.00/332.10/162.80) lat (ms,95%): 707.07 err/s: 0.00 reconn/s: 0.00
SQL statistics:
queries performed:
read: 133924
write: 38264
other: 19132
total: 191320
transactions: 9566 (95.33 per sec.)
queries: 191320 (1906.56 per sec.)
ignored errors: 0 (0.00 per sec.)
reconnects: 0 (0.00 per sec.)
General statistics:
total time: 100.3457s
total number of events: 9566
Latency (ms):
min: 51.94
avg: 335.14
max: 1405.78
95th percentile: 657.93
sum: 3205913.85
Threads fairness:
events (avg/stddev): 298.9375/5.15
execution time (avg/stddev): 100.1848/0.14MySQL container performance monitoring graph.
Clean up the test data (in order to ensure more accurate data, it is recommended to clean up the data, prepare the data, and test before each test).
$ sysbench --db-driver=mysql --mysql-host=192.168.9.91 --mysql-port=32529 --mysql-user=root --mysql-password=P@88w0rd --mysql-db=sbtest --table-size=100000 --tables=16 --threads=32 --events=999999999 --report-interval=10 --time=100 /usr/share/sysbench/oltp_read_write.lua cleanup
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Dropping table 'sbtest1'...
Dropping table 'sbtest2'...
Dropping table 'sbtest3'...
Dropping table 'sbtest4'...
Dropping table 'sbtest5'...
Dropping table 'sbtest6'...
Dropping table 'sbtest7'...
Dropping table 'sbtest8'...
Dropping table 'sbtest9'...
Dropping table 'sbtest10'...
Dropping table 'sbtest11'...
Dropping table 'sbtest12'...
Dropping table 'sbtest13'...
Dropping table 'sbtest14'...
Dropping table 'sbtest15'...
Dropping table 'sbtest16'...Test Results
The results are summarized and compared.
| Number of stress test threads | TPS | QPS | Delay |
|---|---|---|---|
| 8 | 61 | 1221 | 130 |
| 16 | 79 | 1580 | 202 |
| 32 | 95 | 1906 | 335 |
It is recommended to tune according to the test results!
Summarize
This article introduces in detail the common operations of Git, how to store and synchronize code in multiple online code repositories, and also introduces the basic concepts of GitOps and demonstrates how to use the GitOps concept to deploy MySQL services on native K8s. Finally, it demonstrates the installation and basic use of MySQL's common performance testing tool sysbench.
Some of my years of operation and maintenance experience and operation and maintenance ideas run through the whole text.
This article is published by OpenWrite , a multi-post blog platform!
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。