SpringCloud+Docker+Jenkins+GitLab+Maven realizes automated construction and deployment

Foreword and original intention

This article will involve the Docker common commands combined with practical operations in different scenarios. This article will deal with the actual details of the original intention based on the project case scenarios of deploying different environment servers during the work process.

This article focuses on Docker , Jenkins , gitlab , Git , the JDK , SpringBoot , Maven and other technologies to automate the operation and maintenance deployment ( DevOps ) application engineering for SpringCloud deployment .

Original intention: I encountered more interesting problems during the learning process, but spent a little effort and time to sort out, but I sorted out an article that is relatively complete and has a knowledge system of DevOps automated construction and deployment engineering articles, with more technical knowledge content , And the content of the article is longer, but it is divided into several regulations to tell

What is DevOps?

DevOps (combination of Development and Operations) is a collective term for a set of processes, methods, and systems, used to promote communication, collaboration, and collaboration between development (application/software engineering), technical operations, and quality assurance (QA) departments Integration, it is a culture, movement or convention that values communication and cooperation between "software developers (Dev)" and "IT operation and maintenance technicians (Ops)". By automating the processes of "software delivery" and "architecture change", software can be built, tested, and released more quickly, frequently, and reliably. For more information about DevOps, please refer to: awesome! Alibaba Cloud's internal ultra-full K8s/ECS/RDS/OSS/DevOps/DTS actual combat manual is available for download!

Its emergence is due to the increasingly clear realization of the software industry: in order to deliver software products and services on time, development and operations must work closely together.

Involving the content of the software environment

• Centos7 install JDK1.8-u121
• Centos7 install Maven 3.6.1
• Centos7 install Git
• CentOS7 and Git configuration password-free login
• Docker install GitLab
• Docker creates NetWork network
• Docker install Registry private server
• Docker install Jenkins

Special Note

1. How to use Maven and Docker to compile SpringBoot applications into usable images for deployment.

2. JDK and Maven are installed in the traditional way. Since my Centos operating system has other software that depends on them, sometimes it is better to install the software in a traditional way. I will not elaborate here. Some software is not as simple as the traditional way in the Docker installation and use process, such as Jenkins.

What needs to be prepared

Engineering structure

Open IDEA or Eclipse to create a SpringBoot application.

Special instructions for environment configuration

Note: Among them, Gitlab, Registry, and Jenkins are all installed on the node1 machine, that is, node1 is used as the master (master), and node2 is used as the slave (slave or auxiliary). The machine name is meaningful or distinguishable. It is recommended to start with master And slave, I won’t go into too much here. In order to avoid any questions about reading the article, please see the list. In addition, pay attention to the official account of Migrant Workers Technology Road, reply "Java Full Stack Knowledge System", and send you a copy of 40,000 words super Strong summary, Java development full-stack technology knowledge system. PDF manual!

SpringBoot configuration and code detailed explanation

• Pom.xml configuration of the project

SpringBoot and Docker dependent jar configuration

<dependencies>
<!-- Springboot依赖的Jar包 -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

<!-- Springboot热部署jar-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>

<!--yml配置文件提示插件-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-configuration-processor</artifactId>
    <optional>true</optional>
</dependency>

<!-- spring-boot测试jar -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

    </dependencies>
    <build>
        <finalName>springboot</finalName>
        <!-- 一定要声明如下配置 打包xml 到Jar包 -->
        <!-- <resources>
            <resource>
                <directory>src/main/java</directory>
                是否替换资源中的属性
                <filtering>false</filtering>
            </resource>
        </resources>
        <sourceDirectory>${project.basedir}/src/main/java</sourceDirectory>
         -->
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <!-- 默认支持jdk1.8编译 -->
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
<!--docke rmaven编译插件-->
            <plugin>
                <groupId>com.spotify</groupId>
                <artifactId>docker-maven-plugin</artifactId>
                <version>0.4.12</version>
                <configuration>
                    <dockerDirectory>${project.basedir}</dockerDirectory>
                    <resources>
                        <resource>
                            <targetPath>/</targetPath>
                            <directory>${project.build.directory}</directory>
                            <include>${project.build.finalName}.jar</include>
                        </resource>
                    </resources>
                </configuration>
            </plugin>
             <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>com.flong.SpringbootApplication</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin> 
        </plugins>
</build>

• no main manifest attribute error resolution

Configuration project main entrance

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <configuration>
        <archive>
            <manifest>
          <mainClass>com.flong.SpringbootApplication</mainClass>
            </manifest>
        </archive>
    </configuration>
</plugin> 

• env environment variable file

Used to set environment dynamic parameters, the file is in .env format

JAVA_OPTS_DEFAULT=-Xmx512m

• Dockerfile packaging project mirroring in detail

Take the Dockerfile of the development environment as an example. If it is a test environment, change all paths including springboot_dev to springboot_test.

FROM frolvlad/alpine-oraclejdk8:slim
MAINTAINER [e-mail]
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN mkdir -p /home/devsoft/springboot_dev
WORKDIR /home/devsoft/springboot_dev
EXPOSE 7011
ADD ./target/springboot.jar ./
CMD java ${JAVA_OPTS_DEFAULT} -Djava.security.egd=file:/dev/./urandom -jar springboot.jar

Parameter Description

The WORKDIR working directory indicates that when entering the container, there will be a .jar added in the ADD of the Dockerfile

docker exec -it #容器名称或容器id /bin/sh 

Or to use sh and bash, see COMMAND, -it

docker exec -it #容器名称或容器id/bin/bash

• Detailed explanation of the build.sh file shell script

Note 1: The [equal sign] of the tested dynamic variable cannot be set with spaces and tab keys, otherwise the value cannot be obtained, and the space formatting is not supported in the shell script code, and the tab setting formatting is supported. In the terminal (ssh software side) or Jenkins client shell command, the parameters of "location variable" are separated by spaces.

如：sh build.sh 192.168.1.235 springboot 0.0.1 7011 /home/jenkins/workspace/springboot_dev

$IMG_NAME:$IMG_VERSION这个IMG_VERSION版本(tag)参数不指定默认latest

Note 2: Generally, Docker executes Dockerfile by default, but you can customize the suffix file for compilation. The prerequisite must be -f (force) to force the specified file to run

#!/usr/bin/env bash
# 动态变量的【等号】不能有空格和tab键置位,否则获取不了值，而且在shell脚本代码里面不支持空格格式化，支持tab置位格式化。
# 在终端(ssh软件端)或Jenkins客户端shell命令,参数以空格隔开。如：sh build.sh 192.168.1.235 springboot 0.0.1 7011 /home/jenkins/workspace/springboot_dev 
IMG_SERVER="$1"
IMG_NAME="$2"
IMG_VERSION="$3"
IMG_PORT="$4"
RUN_EVN="$5"
IMG_PATH="$6"

echo "服务地址：$IMG_SERVER"
echo "工程镜像名称：$IMG_NAME"
echo "工程版本号：$IMG_VERSION"
echo "工程端口：$IMG_PORT"
echo "服务环境：$RUN_EVN"

#私服访问url路径和编译之后镜像文件存放到指定路径固定,不动态参数进行处理传值.
REGISTRY_URL="192.168.1.235:5000"
IMG_TAR_GZ_PATH="/home/img_tar_gz_path/"

# 判断动态参数不为空字符串的时候才执行下面操作
if [ "$IMG_SERVER" != "" ] && [ "$IMG_NAME" != "" ] && [ "$IMG_VERSION" != "" ] && [ "$IMG_PORT" != "" ]; then
    
   echo " .......进入删除  Container & Images 操作 ......."
    # 清理虚悬镜像,释放磁盘空间
    #docker images|grep none|awk '{print $3 }'|xargs docker rmi

    # 获取容器ID
    CONTAINER_ID=`docker ps -a | grep $IMG_NAME | awk '{ print $1 }'`

    # 获取镜像ID
    IMAGE_ID=`docker images | grep $IMG_NAME | awk '{ print $3 }'`

    # 判断是否存在删除开发容器
    if [[ "$CONTAINER_ID" != "" ]]; then
        docker rm -f $CONTAINER_ID
    fi

    # 判断是否存在删除开发镜像
    if [[ "$IMAGE_ID" != "" ]]; then
        docker rmi -f $IMAGE_ID
    fi
    # $IMG_NAME:$IMG_VERSION 这个IMG_VERSION版本(tag)参数不指定默认latest,通过不同参数执行不同环境文件
    # -f 表示强制指定Dockerfile文件进行编译
    
    echo " .......进入Building & Images 操作 ....... "
    
    #方法1、指定不同文件存放默认的Dockerfile，使用-f进行强制编译
    #docker build -t $IMG_NAME:$IMG_VERSION -f $IMG_PATH"env/"$RUN_EVN/Dockerfile $IMG_PATH
    
    #方法2、跟据不同Dockerfile文件的后缀进行编译不同环境的文件
    docker build -t $IMG_NAME:$IMG_VERSION -f $IMG_PATH"env/"Dockerfile_$RUN_EVN $IMG_PATH
    
    
    # 将镜像打一下标签，然后安照标签进行推送到私服里面，标签名就以服务名即可
    docker tag $IMG_NAME:$IMG_VERSION $REGISTRY_URL/$IMG_NAME:$IMG_VERSION
    
    # 推镜像到私服里面
    docker push $REGISTRY_URL/$IMG_NAME:$IMG_VERSION
    
    # 判断是否存在文件夹
    if [ -d "$IMG_PATH" ];then
         echo "已经存在:"$IMG_PATH
    else
        mkdir -p $IMG_PATH
    fi

    # 保存编译之后镜像文件存放到指定路径
    docker save $IMG_NAME -o $IMG_TAR_GZ_PATH/$IMG_NAME.tar.gz
    
    echo " .......进入Runing操作 ....."
    docker run -d --network default_network --restart=always --env-file=./.env  -e spring.profiles.active=$RUN_EVN --expose=$IMG_PORT --name=$IMG_NAME  -p $IMG_PORT:$IMG_PORT $IMG_NAME:$IMG_VERSION

    echo " .......Build & Run Finish Success~...."
else
    echo " .......Illegal Command Operation ......."
fi

• Use of Docker (save, load, tag, push, pull) commands

Among them, push and pull are one for pushing and the other for pulling. To a certain extent, they both operate on the mirror image on the private server.
The docker save command is to save the compiled tar.gz or tar compressed file, the syntax is as follows:

docker save 镜像名 -o 路径/镜像名.tar.gz

or

docker save 镜像名 -o 路径/镜像名.tar

The docker load command is used to import the image exported using the docker save command. This command is very important. Because some customers require project projects to be deployed on the internal network, this command is reflected when the project is deployed without a network. It has an important position. Grammar docker load [OPTIONS] , the loading process is a bit slow, because the file is a bit large, which shows the Loady Layer [======] , which confirms that the mirroring is a hierarchical relationship.

docker load -i /home/img_tar_gz_path/springboot.tar.gz

Parameter Description

The docker tag and docker push commands are used together, tag first and then push. Each image name and version are distinguished by colons, and docker pull is used according to the situation.

# 将镜像打一下标签，然后安照标签进行推送到私服里面，标签名就以服务名即可
docker tag 镜像名:版本号 私服路径/镜像名:版本号

# 推镜像到私服里面
docker push私服路径/镜像名:版本号

View mirror

The browser verifies the image on the private server by docker push

• Phantom mirror

If the docker compiles unsuccessfully, or the new version overwrites the old version, it will be classified as a dangling mirror. The generated mirror has neither a warehouse name nor a label, which is <none> . Generally speaking, phantom images have lost their value of existence and can be deleted at will. In addition, pay attention to the official account of Migrant Workers Technology Road, reply "Java Full Stack Knowledge System", and send you a 40,000-word super summary, Java Development Full Stack Technical Knowledge System. PDF manual!

• Configuration files for different environments

Configuration parameter

Configuration parameter content of different environments

server:
  port: 7011
runEvn: '开发环境'
server:
  port: 7011
runEvn: '测试环境'
4.2.9 Controller测试代码
@RestController
public class SimpleController {
   //读取配置动态参数
    @Value("${runEvn}")
    private String runEvn;
    
    @GetMapping("/test")
    public String test() {
        return "this spring boot " + runEvn +" date long " 
       + System.currentTimeMillis();
    }
}

Non-multiple machines secret-free remote login & detailed Jenkins deployment

Special Note

Take the development environment as an example to illustrate: the development environment deployment target machine is the same machine as the Jenkins machine. Generally, Jenkins is a separate machine. In order to save its own computer memory, it is placed on the same machine for demonstration and learning.

New maven project

Click on the new task menu of Jenkins

Description of the parametric construction process

• Add parameters

• The parameter description takes the development environment as a case

• Source code management

• Build compilation settings

Indicates that the test unit class is ignored for compilation

clean install -U -Dmaven.test.skip=true

SSH Publishers settings

• Among them, the SSH Server Name is at http://jenkins address: port/jenkins/configure is set to select
• Transfer Set Source file The path of the transfer file, which can be obtained by using the placeholder ${serverPath} constructed by the parameter
• Remote directory remote file directory, similarly also parameterized placeholder ${serverPath} to get

• The SSH Publishers shell script #!/bin/bash tells the terminal to use the bash parser for execution, and only the first line of bash is valid.

  #!/bin/bash
  # 创建目录
  mkdir -p ${serverPath}
  # 切换目录
  cd ${serverPath}
  # 运行脚本
  sh build.sh  $server  ${appName}  ${version}  ${port} ${env} ${serverPath}

  Build and compile the deployment project

The construction parameters of the screenshot are all the parameters configured in the parametric construction process

Build & Run && Deployment Results

• Build process

• Deployment result

Multiple machines secret-free remote login & detailed explanation of Jenkins deployment process

Special Note

Take the test environment as an example to illustrate: the steps are almost the same, the only difference is that the SSH Publishers and the source code storage path are different, and the target machine of the test environment deployment is a different machine from the Jenkins machine.

New maven project

Click on the new task menu of Jenkins

Description of the parametric construction process

• Add parameters

• The parameter description takes the test environment as a case

• Source code management

• Build compilation settings

SSH Publishers settings

• The SSH Server Name is set at http://jenkins address: port/jenkins/configure for selection
• Transfer Set Source file The path of the transfer file, which can be obtained by using the placeholder ${serverPath} constructed by the parameter
• Remote directory remote file directory, similarly also parameterized placeholder ${serverPath} to get
• docker_server1 means that the same host is deployed with Jenkins, and there is no login operation for remote machine shell script operations using scripts.

• The docker_server1 Shell script #!/bin/bash tells the terminal to use the bash parser for execution, and only the first line of bash is valid.

#!/bin/bash
#  打印信息
echo "用户名${userName}"
echo "服务器${server}"
echo "服务器目录${serverPath}"
#  远程创建存放远程上传的代码目录路径
ssh $server  mkdir -p ${targetServerPath}
#  远程拷贝代码到目标机器指定路径
scp -r ${serverPath}/  ${userName}@${server}:${targetServerPath}

docker_server2 indicates the target machine to be deployed, so its script is different from docker_server1.

docker_server2 Shell script

#!/bin/bash
# 切换文件目录
cd ${serverPath}
# 执行脚本
sh build.sh ${server}  ${appName}  ${version}  ${port}  ${env} ${serverPath}

Build and compile the deployment project

The construction parameters of the screenshot are all the parameters configured in the parametric construction process

Build & run & deploy results

• Build process

• Deployment result

Summary & Suggestions

1. This article is for reference only. If you need to learn systematically, you can find information to learn from yourself.

2. The above questions are sorted out and summarized one by one according to the problems encountered by the individual in the actual learning process. In addition to technical problems, many online materials are sorted and shared after learning.

3. There are also many difficulties and doubts in the learning process. If there are any problems or missed points, I hope that the old drivers will point out or make suggestions. I will adopt all kinds of good suggestions and correct ways to continuously improve the current situation. People need high-quality nourishment in the process of growth.

4. When you encounter problems, it is recommended to ask more of the great gods such as "Google, Bing, stackoverflow, and Du Niang".

5. It is recommended to look at the official manual to be more authoritative, because with the development and iteration of technology, the official document is usually updated faster, and the domestic website information is updated slowly.

6. Computer is a subject of "learning by doing". It is not to do it after knowing it, but to do it. Practice more, as the saying goes, practice makes perfect.

7. It is recommended to learn what technology "Know how first, then Know Why", which means to get started first, make a HelloWorld, and then delve into the meaning.

8. I hope this article can help you better understand what (DevOps) is to automate the construction and deployment of images, how to automatically build images and deploy service applications in Docker+Jenkins+GitLab+Maven+SpringBoot&SpringCloud. The entire learning process and construction will be a bit tortuous , It won’t go so smoothly, and I hope you read this article or find information for personal experience and learning effects will be better.

Author: Yinwu Link: jianshu.com/p/bdb0642b7495