Ops.CI/CD-Docker study notes

🎉 Docker Introduction and Installation

What is Docker

Docker is a tool for application packaging, distribution, and deployment. You can also think of it as a lightweight virtual machine. It only virtualizes the running environment your software needs.
A normal virtual machine is a complete and huge system, containing all kinds of software whether you want it or not.

Compared with ordinary virtual machine

characteristic	normal virtual machine	Docker
Cross-platform	Usually only runs on desktop-level systems, such as Windows/Mac, not on servers without a graphical interface	There are many supported systems, all kinds of windows and Linux are supported
performance	High performance loss and high memory usage, because the entire complete system is virtualized	Good performance, only the running environment required by virtual software, minimizing useless configuration
automation	Need to install everything manually	One command can automatically deploy the required environment
stability	The stability is not high, and the difference between different systems is large	Good stability, different systems are deployed in the same way

package, distribute, deploy

Packaging : It is to package the dependencies, third-party libraries, and software required for the operation of your software together into a ==installation package==
Distribution : You can upload your packaged "installation package" to a mirror repository, and others can easily obtain and install it
Deployment : With the "installation package", you can run your application with one command, and automatically simulate the exact same operating environment, whether it is on Windows/Mac/Linux.

Advantages of Docker Deployment

Conventional application development and deployment method: develop and test on Windows by yourself --> configure the running environment deployment on the Linux server.

Question: I have no problem running on my machine, but there are various problems when I get to the server

Use Docker to develop and deploy the process: develop and test on Windows by yourself --> package it as a Docker image (which can be understood as a software installation package) --> only need one command to deploy on various servers

Advantages: It ensures that running on different machines is a consistent operating environment, and there will be no situation where my machine runs normally and your machine runs properly.

what Docker is usually used for

• Application distribution, deployment, easy to spread to others for installation. Especially open source software and applications that offer private deployment
• Quickly install test/learning software, throw it away after use (similar to small programs), and don't waste time installing software. For example Redis/MongoDB/ElasticSearch/ELK
• Multiple versions of software coexist without polluting the system, such as Python2, Python3, Redis4.0, Redis5.0
• Experience/Learn Various Linux Systems on Windows

Key Concepts: Images, Containers

Mirror : It can be understood as a software installation package, which can be easily spread and installed.
Container : After the software is installed, each software running environment is independent and isolated, which is called a container.

Install

Desktop version: https://www.docker.com/products/docker-desktop
Server version: https://docs.docker.com/engine/install/#server

Startup error resolution

Error screenshot

Workaround :
Control Panel->Programs->Enable or Disable Windows Features, Enable Windows Virtualization and Linux Subsystem (WSL2)

Install the Linux subsystem from the command line (this step may not be needed, because the docker installation says to install wsl automatically)
wsl --install -d Ubuntu

The command line installation may be too slow, you can also open the Microsoft Store and search for Linux to install

Setting up the Hypervisor at boot
bcdedit /set hypervisorlaunchtype auto

Be careful to open PowerShell with administrator privileges

Make sure that the BIOS has enabled virtualization, check whether it is enabled as shown below

If it is disabled, please press F2 to enter the BIOS and turn it on. If you cannot set it, you can search for the setting method of your motherboard online. The settings of Intel and AMD may be slightly different.

The following error appears, click the link to install the latest version of WSL2
https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi

Image acceleration source

Mirror Accelerator	Mirror accelerator address
Docker China official image	https://registry.docker-cn.com
DaoCloud mirror site	http://f1361db2.m.daocloud.io
Azure China Mirror	https://dockerhub.azk8s.cn
HKUST Mirror Station	https://docker.mirrors.ustc.edu.cn
Ali Cloud	https://<your_code>.mirror.aliyuncs.com
Seven Niuyun	https://reg-mirror.qiniu.com
NetEase Cloud	https://hub-mirror.c.163.com
Tencent Cloud	https://mirror.ccs.tencentyun.com

💻 Docker Quick Install Software

Disadvantages of direct installation

• Installation is troublesome, there may be various dependencies, and an error is reported when running. For example: WordPress, ElasticSearch, Redis, ELK
• It may not be friendly to Windows, there are various compatibility problems in running, the software only supports running on Linux
• It is inconvenient to install multiple versions of software and cannot coexist.
• The computer has a bunch of software installed, which slows down the computer.
• Different systems and hardware, different installation methods

Advantages of Docker installation

• One command can be installed, fast and convenient
• There are a large number of mirrors, which can be used directly
• No system compatibility issues, Linux exclusive software still runs
• Support software multi-version coexistence
• Throw it away after use, don't slow down your computer
• Different systems and hardware, as long as Docker is installed, everything else is the same, one command can do everything

Demo Docker install Redis

Redis official website: https://redis.io/

The official website download and installation tutorial only has the source code installation method, and there is no Windows version. If you want to install the windows version yourself, you need to find the installation package compiled by others.

Find Redis in the official Docker image repository: https://hub.docker.com/

A command to run: docker run -d -p 6379:6379 --name redis redis:latest
Command reference: https://docs.docker.com/engine/reference/commandline/run/

Install ELK

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

Insufficient memory solution
Go to the user directory cd ~ , the path is similar to this: C:\Users\<UserName>
Create .wslconfig file fill in the following content

 [wsl2]
memory=10GB # Limits VM memory in WSL 2 to 4 GB
processors=2 # Makes the WSL 2 VM use two virtual processors

The configuration takes effect, and the command line runs wsl --shutdown

More related commands

docker ps View currently running containers
docker images View mirror list
docker rm container-id Delete the container with the specified id
docker stop/start container-id stop/start the container with the specified id
docker rmi image-id Delete the mirror with the specified id
docker volume ls View volume list
docker network ls View network list

💽 Make your own mirror

Build images for your own web projects

Example project code: https://github.com/gzyunke/test-docker
This is a Web project written by Nodejs + Koa2, which provides two simple demo pages.
Software dependencies: nodejs
Project dependencies: koa, log4js, koa-router

Write Dockerfile

 FROM node:11
MAINTAINER easydoc.net

# 复制代码
ADD . /app

# 设置容器启动后的默认运行目录
WORKDIR /app

# 运行命令，安装依赖
# RUN 命令可以有多个，但是可以用 && 连接多个命令来减少层级。
# 例如 RUN npm install && cd /app && mkdir logs
RUN npm install --registry=https://registry.npm.taobao.org

# CMD 指令只能一个，是容器启动后执行的命令，算是程序的入口。
# 如果还需要运行其他命令可以用 && 连接，也可以写成一个shell脚本去执行。
# 例如 CMD cd /app && ./start.sh
CMD node app.js

Dockerfile documentation

Practical tips:
If you often encounter some running errors, dependency errors, etc. when writing Dockerfile, you can directly run a dependent base, then enter the terminal to configure the environment, and then write the completed steps in the Dockerfile after success, so that you can write debugging will be much faster.
For example, the bottom of the above is node:11 , we can run docker run -it -d node:11 bash , after running, enter the container terminal to configure the dependent software, then try to run our own software, and finally put all the steps that have been done. Just write to Dockerfile.
Master this skill, and your Dockerfile will be very handy to write.

Build as image (install package) and run

Compile docker build -t test:v1 .

-t Set image name and version number command reference: https://docs.docker.com/engine/reference/commandline/build/

run docker run -p 8080:8080 --name test-hello test:v1

-p Map the port in the container to the host
--name Container name
-d Background run command reference documentation: https://docs.docker.com/engine/reference/run/

More related commands

docker ps View currently running containers
docker images View mirror list
docker rm container-id Delete the container with the specified id
docker stop/start container-id stop/start the container with the specified id
docker rmi image-id Delete the mirror with the specified id
docker volume ls View volume list
docker network ls View network list

🥙Directory mount

Existing Problems

• After using Docker to run, we changed the project code and it will not take effect immediately, we need to re build and run , which is very troublesome.
• The data generated in the container, such as log files and database backup files, are lost after the container is deleted.

Several mounting methods

• bind mount Directly map the host directory to the container, suitable for hanging code directories and configuration files. Can be attached to multiple containers
• volume Created and managed by the container, created on the host machine, so deleting the container will not be lost, officially recommended, more efficient, Linux file system, suitable for storing database data. Can be attached to multiple containers
• tmpfs mount Suitable for storing temporary files in the host memory. Multiple containers cannot be shared.

Documentation reference: https://docs.docker.com/storage/

mount demo

bind mount method with absolute path -v D:/code:/app

volume way, only need one name -v db-data:/app

Example:
docker run -p 8080:8080 --name test-hello -v D:/code:/app -d test:v1

Notice!
Because after mounting, the code in the container will be replaced with the code of your local machine. If your code directory does not have the node_modules directory, you need to execute it in the code directory npm install --registry=https://registry.npm.taobao.org to ensure dependencies The library has been installed, otherwise it may prompt "Error: Cannot find module 'koa'"
If your computer does not have nodejs installed, you need to install it to execute the above command.

👨‍👦‍👦Multi-container communication

learning target

Projects are often not run independently, and require databases and caches to work together.
In this section, we add a Redis dependency to the previous Web project, run one more Redis container, and demonstrate how to communicate between multiple containers.

Create a virtual network

To communicate between multiple containers and access the Redis container from the web container, we only need to put them in the same network.

Documentation reference: https://docs.docker.com/engine/reference/commandline/network/

demo

Create a network named test-net :

docker network create test-net

Run Redis in the test-net network, alias redis

docker run -d --name redis --network test-net --network-alias redis redis:latest

Modify the address of access redis in the code to be a network alias

Run the web project, using the same network

docker run -p 8080:8080 --name test -v D:/test:/app --network test-net -d test:v1

View data

http://localhost:8080/redis
The container terminal checks whether the data is consistent

More related commands

docker ps View currently running containers
docker images View mirror list
docker rm container-id Delete the container with the specified id
docker stop/start container-id stop/start the container with the specified id
docker rmi image-id Delete the mirror with the specified id
docker volume ls View volume list
docker network ls View network list

🍁Docker-Compose

Existing Problems

In the previous section, we ran two containers: Web Project + Redis
If the project relies on more third-party software, we need to manage more containers, each of which must be configured and run separately, specifying the network.
In this section, we use docker-compose to assemble multiple services of the project and run them with one click.

Install Docker Compose

• If you are installing the desktop version of Docker, no additional installation is required, it is already included.
• If it is a server version of Docker without a graphical interface, you need to install the installation documentation separately
• Run docker-compose check if the installation is successful

write scripts

To assemble multiple services that the project depends on, we need to write a docker-compose.yml file to describe which services are depended on. Reference documentation: https://docs.docker.com/compose/

 version: "3.7"

services:
  app:
    build: ./
    ports:
      - 80:8080
    volumes:
      - ./:/app
    environment:
      - TZ=Asia/Shanghai
  redis:
    image: redis:5.0.13
    volumes:
      - redis:/data
    environment:
      - TZ=Asia/Shanghai

volumes:
  redis:

The default time of the container is not Beijing time, adding TZ=Asia/Shanghai can be changed to Beijing time

start running

In docker-compose.yml ==the directory where the file is located==, execute: docker-compose up to run.
Command reference: https://docs.docker.com/compose/reference/up/

Only need to add a -d parameter to run in the background docker-compose up -d
Check the running status: docker-compose ps
Stopped running: docker-compose stop
Reboot: docker-compose restart
Restart a single service: docker-compose restart service-name
Enter the container command line: docker-compose exec service-name sh
View the container running log: docker-compose logs [service-name]

🚚Publish and deploy

Mirror warehouse introduction

The mirror warehouse is used to store the "installation package" we build. Docker officially provides a mirror library , which contains a large number of mirrors. Basically, all kinds of software need dependencies. You can search directly for whatever you want.

We can also upload the image we built to the image library provided by docker for easy dissemination.
Of course, you can also build your own private image library, or use the image hosting services provided by various domestic manufacturers, such as Alibaba Cloud, Tencent Cloud

upload our image

• First you need to register an account
• Create a mirror library
  
• Command line login account:
  docker login -u username
• Create a new tag, the name must be the same as your registered account
  docker tag test:v1 username/test:v1
• push up
  docker push username/test:v1
• try deployment
  docker run -dp 8080:8080 username/test:v1

This image can also be used directly in docker-compose

 version: "3.7"

services:
  app:
#    build: ./
    image: helloguguji/test:v1
    ports:
      - 80:8080
    volumes:
      - ./:/app
    environment:
      - TZ=Asia/Shanghai
  redis:
    image: redis:5.0.13
    volumes:
      - redis:/data
    environment:
      - TZ=Asia/Shanghai

volumes:
  redis:

Alibaba Cloud Container Hosting

Docker's official image hosting is sometimes too slow to upload and download. If you want faster speed, you can use Alibaba Cloud's free image hosting to log in to Alibaba Cloud

🎯 Backup and migrate data

Introduction to Migration

If the data in the container is not used to mount the directory, the data will be lost after the container is deleted.
We have already explained how to mount directories
If you use bind mount volume directly mount the host's directory into the container, it is very convenient to migrate data, just copy the directory directly. The data is created and managed by the container, and it needs a special way to get the data out.

Process for backing up and importing volumes

Backup:

• Run an ubuntu container, mount the volume to be backed up to the container, and mount the host directory to the backup directory in the container.
• Run the tar command to compress the data into a file
• Copy the backup file to the machine that needs to be imported

import:

• Run the ubuntu container, mount the volume of the container, and mount the directory where the host backup file is located to the container
• Run the tar command to extract the backup file to the specified directory

Backup MongoDB data demo

• Run a mongodb and create a volume named mongo-data pointing to the container's /data directory
  docker run -p 27018:27017 --name mongo -v mongo-data:/data -d mongo:4.4
• Run an Ubuntu container, mount mongo all volumes of the container, map the backup directory of the host to the /backup directory in the container, and then run the tar command to compress and package the data
  docker run --rm --volumes-from mongo -v d:/backup:/backup ubuntu tar cvf /backup/backup.tar /data/

Finally, you can take this backup.tar file and import it elsewhere.

Restore Volume Data Demo

• Run an ubuntu container, mount all volumes of the mongo container, then read the backup files in the /backup directory and extract them to the /data/ directory
  docker run --rm --volumes-from mongo -v d:/backup:/backup ubuntu bash -c "cd /data/ && tar xvf /backup/backup.tar --strip 1"

Note that volumes-from specifies the container name
strip 1 means to remove the first layer of directories when decompressing, because the absolute path is included when compressing