Build a CICD environment from 0 to 1 based on Jenkins+Maven+Gitea+Nexus

In the traditional monolithic software architecture, software development, testing, and operation and maintenance are all based on a single process.

When split into microservices, a single application can be split into multiple microservices, such as user systems, which can be split into multiple microservice modules such as basic information management, credit management, order management, user information management, and contract management .

At this time, if each module is packaged, released, run, developed, tested, and operated separately, the workload of testing and operation and maintenance will be greatly increased.

In this process, if there is a lack of capabilities such as automated testing, automated integration/deployment, and automated operation and maintenance, the impact will be

Increased software delivery cycles
In the case of multi-environment deployment, problems caused by differences in each environment.
Manual operation and maintenance is easy to bring some irreproducible effects to the environment, and once an operation and maintenance error occurs, it is difficult to recover immediately, resulting in a long troubleshooting time. And higher requirements for the ability of operation and maintenance personnel

All of these issues lead to longer software delivery times, increased risk, and increased operational and maintenance costs. Therefore, we need an automated deployment system to build a CICD model.

How ordinary Jar packages work

Use maven package
nohup java -jar ${APP_NAME} > goods-service.log 2>&1 &

nohup purpose: run commands without hanging up
& uses, runs in the background
2>&1: In bash:
- 0 represents STDIN_FILENO standard input (usually the keyboard),
- 1 stands for STDOUT_FILENO standard output (usually the display screen, to be precise the user terminal console),
- 2 and three represent STDERR_FILENO (standard error (error message output).
2>&1 is used to redirect standard error 2 to standard output 1. The & in front of 1 here is to make bash interpret 1 as standard output instead of file 1. As for the last &, it makes bash execute in the background.
> Generate the content directly to the specified file

Build a Nexus private server environment

Nexus is a powerful Maven repository manager that greatly simplifies the maintenance of local internal repositories and access to external repositories. Nexus is an "out of the box" system that does not require a database, it uses a file system plus Lucene to organize data.

Maven private server environment needs to use sonatype nexus , let's make a detailed analysis from installation and configuration below

Deployment server: 192.168.8.138

Download and install

Visit: https://sonatype-download.global.ssl.fastly.net/repository/downloads-prod-group/3/nexus-3.37.0-01-unix.tar.gz address to download Sonatype Nexus.
Unzip to /data/program directory

[root@localhost program]# tar -zxvf nexus-3.37.0-01-unix.tar.gz

${NEXUS_HOME}\bin to the 061e1349603178 directory and execute the following command to start Nexus

./nexus start

Install Maven

Download Maven: https://dlcdn.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz
Configure Maven and JDK environment variables.

export JAVA_HOME=/data/program/jdk1.8.0_241
export MAVEN_HOME=/data/program/apache-maven-3.8.4
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

${NEXUS_HOME}\bin to the 061e134960325f directory and execute the following command to start Nexus
Start with ./nexus start, start in the background, and you can access it after successful startup
Start with ./nexus run, the foreground starts, the log is displayed, and it can be accessed after startup

./nexus start

When starting, there will be the following prompt. This is to suggest that we use a non- root account to access.

WARNING: ************************************************************
WARNING: Detected execution as "root" user.  This is NOT recommended!
WARNING: ************************************************************

Visit: http://localhost:8081 Visit Nexus Repository

If you want to configure the JVM parameter that the nexus application starts locally, you can nexus.vmoptions

If you want to change the port number of , you can use nexus-default.properties

log in to the console

The default login account is admin , and the password will prompt you to be in: /data/program/sonatype-work/nexus3/admin.password file.

The content is as follows, just copy the content and log in.

090849ac-cea7-4353-b2c8-59b2bceadb50

Nexus Console Instructions

Entering the Browse menu of the Nexus console, you can see four warehouse types:

1) maven-central: maven central library, pulls jar https://repo1.maven.org/maven2/

2) maven-releases: private library release jar

3) maven-snapshots: private library snapshot (debug version) jar

4) maven-public: Warehouse grouping, combining the above three warehouses to provide services to the outside world, and using them in the local maven basic configuration settings.xml.

The default warehouse types of Nexus are as follows: (The above name can be chosen arbitrarily, the key is what warehouse type it corresponds to)

1) group (warehouse group type): also called group warehouse, which is used to facilitate the warehouse set by developers themselves;

2) hosted (host type): the release warehouse of the internal project (internal developers, the warehouse released and stored);

3) proxy (proxy type): Find the warehouse of data from the remote central warehouse (you can click the Configuration page of the corresponding warehouse to sign the value of the Remote Storage Location property, that is, the path of the remote warehouse being proxied);

4) virtual (virtual type): virtual warehouse (this is basically not used, focus on the use of the above three warehouses);

Nuget is a package manager for Microsoft's .NET development platform, similar to Maven.

Directory Description

nexus-3.34.0-01 directory

bin contains nexus startup scripts and related configurations
Configuration files such as jetty, karaf, etc.
jre jre environment
lib java rack package library
public About the resources required for the nexus application to run locally
system applies all plugins and components
LICENSE.txt and NOTICE.txt Copyright Notice and Legal Details

sonatype-work\nexus3 directory

blobs/ The default path for creating blobs, of course, it can also be re-specified
cache/ Information about currently cached karaf packages
db/ OrientDB database data, the database used to store nexus metadata
elasticsearch/ The currently configured Elasticsearch state
etc/ is presumably related to runtime configuration state and customizations about the repository
health-check/ Look at the directory, the storage directory of the related reports of health check.
keystores/ auto-generated ID primary key about the repository
log/ The log files generated by the running instance, as well as the compressed package of the log files, seem to be generated every day, you can delete the old log files regularly
tmp/ directory for storing temporary files

Nexus is set as a system service

Follow the steps below

Modify ${NEXUS_HOME}\bin\nexus and add the following configuration

INSTALL4J_JAVA_HOME_OVERRIDE=/data/program/jdk1.8.0_241

set soft link

[root@localhost bin]# ln -s /data/program/nexus-3.37.0-01/bin/nexus /etc/init.d/nexus

Configure system services through chkconfig

cd /etc/init.d
sudo chkconfig --add nexus #添加nexus服务
sudo chkconfig --levels 345 nexus on #设置开启自启动

Start and stop services
sudo service nexus start #Start the service
service nexus status #View service status

Setting up the Gitea environment

Reference document: https://docs.gitea.io/zh-cn/install-from-binary/

Install git environment: yum -y install git .
Download the installation package in linux to the /data/program/gitea directory through the following command.

wget -O gitea https://dl.gitea.io/gitea/1.15.7/gitea-1.15.7-linux-amd64

Execute the chmod +x gitea command and grant execute permission
Execute the following command to run gitea

./gitea web

Install as a system service (important)

Create a Git user

sudo useradd \
   --system \
   --shell /bin/bash \
   --comment 'Git Version Control' \
   --create-home \
   --home /home/git \
   git

Download binaries

wget -O gitea https://dl.gitea.io/gitea/1.15.7/gitea-1.15.7-linux-amd64

According to the official recommendation of gitea, configure the installation directory of gitea as follows

Move the downloaded file to the /usr/local/bin directory
```
sudo mv /data/program/gitea /usr/local/bin
```
Make the binary executable:
```
chmod +x /usr/local/bin/gitea
```

Follow the commands below to create the necessary directories and set permissions

sudo mkdir -p /var/lib/gitea/{custom,data,indexers,public,log}
sudo mkdir -p /var/lib/gitea/{custom,data,indexers,public,log}
sudo chown git: /var/lib/gitea/{data,indexers,custom,public,log}
sudo chmod 750 /var/lib/gitea/{data,indexers,log}
sudo mkdir /etc/gitea
sudo chown root:git /etc/gitea
sudo chmod 770 /etc/gitea

Configure system services according to the Systemd Unit file officially provided by Gitea.
```
sudo wget https://raw.githubusercontent.com/go-gitea/gitea/master/contrib/systemd/gitea.service -P /etc/systemd/system/
```
Note that gitea.service cannot be downloaded through wget and needs to be copied on github
After completing the above process, enable automatic startup with the following command
```
systemctl enable gitea
systemctl start gitea
```

After the installation is started, visit: http://192.168.8.136:3000 to configure the database related properties.

Build the Jenkins environment

Jenkins is an open source continuous integration tool written in JAVA, running in a servlet container, supporting software configuration management (SCM) tools, and can execute projects based on APACHE ANT and APACHE MAVEN, as well as arbitrary Shell scripts and Windows batch commands

Jenkins provides automatic build and deployment functions. The specific installation methods are as follows:

wget -O /etc/yum.repos.d/jenkins.repo \
    https://pkg.jenkins.io/redhat-stable/jenkins.repo
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
yum upgrade
yum install epel-release java-11-openjdk-devel
yum install jenkins
systemctl daemon-reload

Start or stop jenkins by following command

systemctl start jenkins
systemctl stop jenkins

Visit: http://192.168.8.136:8080 visit jenkins

Follow the steps prompted by the console step by step.

Note: Jenkins installation uses the JENKINS user by default, so if you use root privileges, you need to modify the account

[root@localhost bin]# vim /etc/sysconfig/jenkins

JENKINS_USER="root"

Project renovation

Project configuration local private server

Modify the setting.xml file and add the mirror configuration
```
<mirrors>
    <mirror>
        <id>nexus</id>
        <mirrorOf>maven-public</mirrorOf>
        <url>http://192.168.8.136:8081/repository/maven-public/</url>
    </mirror>
</mirrors>
```
Mirror is equivalent to an interceptor, which intercepts maven's related requests to the remote repository, and uses the mirror to obtain the jar package.

Add the following configuration to the project, that is, specify the release repository of the jar package of different release versions of snapshots and releases

<distributionManagement>
    <snapshotRepository>
        <id>snapshots</id>
        <name>Nexus Snapshot Repository</name>
        <url>http://192.168.8.136:8081/repository/maven-snapshots/</url>
    </snapshotRepository>
    <repository>
        <id>releases</id>
        <name>Nexus Release Repository</name>
        <url>http://192.168.8.136:8081/repository/huhy-nexus/</url>
    </repository>
</distributionManagement>

Modify the settings.xml file of the publisher

The purpose of modifying the settings.xml file of the publishing server is that when the Jenkins server performs continuous integration, it needs to download the dependent jar package through maven, and this download needs to be obtained from our local private server.

<mirrors>
    <mirror>
        <id>nexus</id>
        <mirrorOf>maven-public</mirrorOf>
        <url>http://192.168.8.136:8081/repository/maven-public/</url>
    </mirror>
</mirrors>

<profiles>
     <profile>
      <id>nexusRep</id>
      <repositories>
        <repository>
          <id>nexus</id>
          <url>http://192.168.8.136:8181/repository/maven-public/</url>
          <layout>default</layout>
          <releases>
              <enabled>true</enabled>
              <updatePolicy>always</updatePolicy>
          </releases>
        </repository>
      </repositories>
      <pluginRepositories>  
        <pluginRepository>  
          <!--插件地址--> 
            <id>nexus</id>  
            <url>http://192.168.8.136:8181/repository/maven-public/</url>  
            <snapshots>     
              <enabled>true</enabled>    
            </snapshots>    
            <releases>      
                <enabled>true</enabled>    
            </releases> 
        </pluginRepository>
       </pluginRepositories>
    </profile>
  </profiles>

  <activeProfiles>
    <activeProfile>nexusRep</activeProfile>
  </activeProfiles>

Configure automatic integration and publishing

Configure Jenkins environment variables

Go to the following page, the configuration marked in red.

Configure the Maven environment and specify the Maven directory installed on the publishing server.

Install the Jenkins plugin

Gitea, integrate Gitea, after installation, add Gitea Server information in Jenkins global configuration.
Git Parameter , configure Git publish properties
Publish Over SSH, execute the script on the remote machine, this step needs to configure the remote machine capable of ssh
Maven Integration, which supports the integration of Maven projects

Configure publish target server information

We treat the following two servers as web nodes

192.168.8.134
192.168.8.135

In the global configuration of Jenkins, configure the information of these two servers for subsequent remote transmission of jar packages. Among them, Remote Directory the working directory of the target server, and the jar package will be remotely transferred to this directory

Add project release mechanism

The task of creating a Maven project.

Configure the source code source, here use the source code address of the project in Gitea, and configure the login account password information.

Configure the execution command of Maven, among which root POM , if it is in a multi-module project, you need to specify the pom.xml of the module to be built currently.

Adding the execution logic after the successful build is to publish the jar package to the remote target server, and then execute the relevant shell script to start the service

Write a release script

Write a shell script, and execute the following script when the jar package is sent to the target server.

Do historical jar backup and cleanup
Execute shell script to start service

#! bin/sh -e
export JAVA_HOME=/data/program/jdk1.8.0_241
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=${PATH}:$JAVA_HOME/bin
source /etc/profile 

# define property
JAR_PATH='/app/service/goods-service'
TEMP_PATH='/app/service/temp'
BACKUP_PATH='/app/service/backup'
JAR_NAME=goods-service.jar
FILE_NAME=goods-service

# stop target service
cd ${JAR_PATH}
sh run-goods-service.sh stop
sleep 2
rm -rf $FILE_NAME.log

# backup old jar
BACKUP_DATE=$(date +%Y%m%d_%H%M)

if [ ! -d $JAR_PATH/backup/$FILE_NAME ];then
    mkdir -p $JAR_PATH/backup/$FILE_NAME
fi

cd ${JAR_PATH}
pwd
if [ -f $JAR_NAME ];then
   mv -f ./$JAR_NAME ./backup/$FILE_NAME/$JAR_NAME$BACKUP_DATE
   sleep 1
fi

# start jar
BUILD_ID=dontKillMe
cd ${TEMP_PATH}
mv -f $JAR_NAME  $JAR_PATH
cd ${JAR_PATH}
sh run-goods-service.sh restart

# clear old backup
cd ${JAR_PATH}/backup/$FILE_NAME
ls -lt|awk 'NR>5{print $NF}' |xargs rm -rf

ps -ef|grep java
echo "=============deploy success========"

Write the run script run-goods-service.sh

# 表示当前脚本采用/bin路径的bash程序来解释执行
#!/bin/bash

# 执行的jar包
APP_NAME=goods-service.jar

usage() {
  echo "执行操作命令 [start|stop|restart|status]"
  exit 1
}

if_exist() {
  pid=`ps -ef|grep $APP_NAME|grep -v grep|awk '{print $2}'`
  if [ -z "${pid}" ]; then
    return 1
  else
    return 0
  fi
}

start() {
  if_exist
  if [ $? -eq 0 ]; then
    echo "${APP_NAME} already running . pid=${pid}"
  else
    nohup java -jar ${APP_NAME} > goods-service.log 2>&1 &
    npid=`ps -ef|grep $APP_NAME|grep -v grep|awk '{print $2}'`
    echo "start ${APP_NAME} success, pid=${npid}"
  fi
}

stop() {
  if_exist
  if [ $? -eq 0 ]; then
    kill -9 $pid
    echo "stop $pid success".
  else
    echo "${APP_NAME} is not running"
  fi
}

status() {
  if_exist
  if [ $? -eq 0 ]; then
    echo "${APP_NAME} is running. pid is ${pid}"
  else
    echo "${APP_NAME} is not running "
  fi
}

restart() {
  stop
  sleep 5
  start
}

case "$1" in
   "start")
      start
       ;;
    "stop")
      stop
       ;;
    "status")
      status
       ;;
    "restart")
      restart
       ;;
    *)
    usage
     ;;
esac

Dynamic build after configuration code is submitted

If we want the code to be submitted and merged into a branch, it will be automatically built and released, how can we do it?

Install the Webhook plugin

Generic Webhook Trigger plugin in Jenkins. After the installation is successful, an option shown below will be added to the build configuration page.
Configure Generiac Webhook Trigger and add a token for verification.
Pay attention to this address: http://JENKINS_URL/generic-webhook-trigger/invoke , you need to configure this as a trigger call in the webhook.

gitea add webhook hook

In gitea's project, find the web hook and add the web hook. Select gitea .
add webhook

`Validate auto-triggered behavior`

Modify any code of the gpmall-pc project and submit it to gitea.
Looking at Jenkin's project build directory, an automatic build task will be added, as shown in the following figure.
And in gitea's webhook, you can see the latest push records

`Source address`

The source code used in the article demo: https://github.com/2227324689/spring-cloud-netflix-example.git

Copyright notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless otherwise stated. Reprint please indicate from Mic takes you to learn architecture! If this article is helpful to you, please help to follow and like, your persistence is the driving force for my continuous creation. Welcome to follow the WeChat public account of the same name to get more technical dry goods!