Java Edition Face Detection Detailed Explanation Part II: Coding

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

Overview of this article

• If you have read "Three-Minute Speed Experience: Face Detection in Java Edition" , and even hands-on the actual operation, you should be interested in the technical details behind it. To develop such an application, we have to do the following three things in total :
• Prepare the docker base image
• Develop java applications
• Package the java application into a package file and integrate it into the basic image to get the final java application image
• For the work of <font color="blue">preparing the docker base image</font>, let’s use the previous article : Docker image of the operating environment (CentOS+JDK+OpenCV)" It's done, the next thing to do is to develop a java application and make it into a docker image

Version Information

• The version information involved in this java application is as follows:
• springboot：2.4.8
• javacpp：1.4.3
• javacv：1.4.3

Source download

• The complete source code of this actual combat can be downloaded on GitHub. The address and link information are shown in the following table ( https://github.com/zq2599/blog_demos):

name	Link	Remark
Project homepage	https://github.com/zq2599/blog_demos	The project's homepage on GitHub
git warehouse address (https)	https://github.com/zq2599/blog_demos.git	The warehouse address of the source code of the project, https protocol
git warehouse address (ssh)	[e-mail]:zq2599/blog_demos.git	The warehouse address of the source code of the project, ssh protocol

• There are multiple folders in this git project. The source code of this article is under the <font color="blue">javacv-tutorials</font> folder, as shown in the red box below:

coding

• In order to unify the management of source code and jar dependencies, the project adopts a maven parent-child structure. The parent project is named <font color="blue">javacv-tutorials</font>. Its pom.xml is as follows. It can be seen that some jar versions are mainly defined :

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.bolingcavalry</groupId>
    <artifactId>javacv-tutorials</artifactId>
    <packaging>pom</packaging>
    <version>1.0-SNAPSHOT</version>
    <modules>
        <module>face-detect-demo</module>
    </modules>

    <properties>
        <java.version>1.8</java.version>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <maven-compiler-plugin.version>3.6.1</maven-compiler-plugin.version>
        <springboot.version>2.4.8</springboot.version>

        <!-- javacpp当前版本 -->
        <javacpp.version>1.4.3</javacpp.version>
        <!-- opencv版本 -->
        <opencv.version>3.4.3</opencv.version>
        <!-- ffmpeg版本 -->
        <ffmpeg.version>4.0.2</ffmpeg.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.projectlombok</groupId>
                <artifactId>lombok</artifactId>
                <version>1.18.18</version>
            </dependency>

            <dependency>
                <groupId>org.bytedeco</groupId>
                <artifactId>javacv-platform</artifactId>
                <version>${javacpp.version}</version>
            </dependency>
            <dependency>
                <groupId>org.bytedeco</groupId>
                <artifactId>javacv</artifactId>
                <version>${javacpp.version}</version>
            </dependency>
            <!-- javacpp -->
            <dependency>
                <groupId>org.bytedeco</groupId>
                <artifactId>javacpp</artifactId>
                <version>${javacpp.version}</version>
            </dependency>
            <!-- ffmpeg -->
            <dependency>
                <groupId>org.bytedeco.javacpp-presets</groupId>
                <artifactId>ffmpeg-platform</artifactId>
                <version>${ffmpeg.version}-${javacpp.version}</version>
            </dependency>
            <dependency>
                <groupId>org.bytedeco.javacpp-presets</groupId>
                <artifactId>ffmpeg</artifactId>
                <version>${ffmpeg.version}-${javacpp.version}</version>
            </dependency>
        </dependencies>

    </dependencyManagement>
</project>

• Create a new sub-project named <font color="red">face-detect-demo</font> under <font color="blue">javacv-tutorials</font>, which is the application we will develop today , Its pom.xml is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>javacv-tutorials</artifactId>
        <groupId>com.bolingcavalry</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>face-detect-demo</artifactId>
    <packaging>jar</packaging>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${springboot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <!--FreeMarker模板视图依赖-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-freemarker</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacv-platform</artifactId>
        </dependency>
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacv</artifactId>
        </dependency>
        <!-- javacpp -->
        <dependency>
            <groupId>org.bytedeco</groupId>
            <artifactId>javacpp</artifactId>
        </dependency>
        <!-- ffmpeg -->
        <dependency>
            <groupId>org.bytedeco.javacpp-presets</groupId>
            <artifactId>ffmpeg-platform</artifactId>
        </dependency>
        <dependency>
            <groupId>org.bytedeco.javacpp-presets</groupId>
            <artifactId>ffmpeg</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- 如果父工程不是springboot，就要用以下方式使用插件，才能生成正常的jar -->
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <mainClass>com.bolingcavalry.facedetect.FaceDetectApplication</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

• The configuration file is as follows, focusing on the configuration of the previous template, file upload size, model file directory, etc.:

### FreeMarker 配置
spring.freemarker.allow-request-override=false
#Enable template caching.启用模板缓存。
spring.freemarker.cache=false
spring.freemarker.check-template-location=true
spring.freemarker.charset=UTF-8
spring.freemarker.content-type=text/html
spring.freemarker.expose-request-attributes=false
spring.freemarker.expose-session-attributes=false
spring.freemarker.expose-spring-macro-helpers=false
#设置面板后缀
spring.freemarker.suffix=.ftl

# 设置单个文件最大内存
spring.servlet.multipart.max-file-size=100MB
# 设置所有文件最大内存
spring.servlet.multipart.max-request-size=1000MB
# 自定义文件上传路径
web.upload-path=/app/images
# 模型路径
opencv.model-path=/app/model/haarcascade_frontalface_default.xml

• The front-end page file has only one <font color="blue">index.ftl</font>. Please forgive Xinchen's influential front-end level. The front-end has only one page, which can be submitted and is also a page that displays the processing results:

<!DOCTYPE html>
<head>
    <meta charset="UTF-8" />
    <title>图片上传Demo</title>
</head>
<body>
<h1 >图片上传Demo</h1>
<form action="fileUpload" method="post" enctype="multipart/form-data">
    <p>选择检测文件: <input type="file" name="fileName"/></p>
    <p>周围检测数量: <input type="number" value="32" name="minneighbors"/></p>
    <p><input type="submit" value="提交"/></p>
</form>
<#--判断是否上传文件-->
<#if msg??>
    <span>${msg}</span><br><br>
<#else >
    <span>${msg!("文件未上传")}</span><br>
</#if>
<#--显示图片，一定要在img中的src发请求给controller，否则直接跳转是乱码-->
<#if fileName??>
<#--<img src="/show?fileName=${fileName}" style="width: 100px"/>-->
<img src="/show?fileName=${fileName}"/>
<#else>
<#--<img src="/show" style="width: 200px"/>-->
</#if>
</body>
</html>

• Let's look at the background code again, first of all the most common application startup classes:

package com.bolingcavalry.facedetect;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class FaceDetectApplication {

    public static void main(String[] args) {
        SpringApplication.run(FaceDetectApplication.class, args);
    }
}

• After the front end uploads the picture, what processing should the back end do? Without posting the code, let's go through the things to be done on the backend, as shown in the following figure:

• Next is the core business class <font color="blue">UploadController.java</font>. The web interface and business logic processing are all in it. They are executed in the order of the process shown in the figure above. There are a few things to pay attention to. The place will be mentioned later:

package com.bolingcavalry.facedetect.controller;

import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.ResourceLoader;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.multipart.MultipartFile;

import java.io.File;
import java.io.IOException;
import java.util.Map;
import org.opencv.core.*;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;
import org.opencv.objdetect.CascadeClassifier;

import java.util.UUID;

import static org.bytedeco.javacpp.opencv_objdetect.CV_HAAR_DO_CANNY_PRUNING;

@Controller
@Slf4j
public class UploadController {

    static {
        // 加载 动态链接库
        System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    }

    private final ResourceLoader resourceLoader;

    @Autowired
    public UploadController(ResourceLoader resourceLoader) {
        this.resourceLoader = resourceLoader;
    }

    @Value("${web.upload-path}")
    private String uploadPath;

    @Value("${opencv.model-path}")
    private String modelPath;

    /**
     * 跳转到文件上传页面
     * @return
     */
    @RequestMapping("index")
    public String toUpload(){
        return "index";
    }

    /**
     * 上次文件到指定目录
     * @param file 文件
     * @param path 文件存放路径
     * @param fileName 源文件名
     * @return
     */
    private static boolean upload(MultipartFile file, String path, String fileName){
        //使用原文件名
        String realPath = path + "/" + fileName;

        File dest = new File(realPath);

        //判断文件父目录是否存在
        if(!dest.getParentFile().exists()){
            dest.getParentFile().mkdir();
        }

        try {
            //保存文件
            file.transferTo(dest);
            return true;
        } catch (IllegalStateException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
            return false;
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
            return false;
        }
    }

    /**
     *
     * @param file 要上传的文件
     * @return
     */
    @RequestMapping("fileUpload")
    public String upload(@RequestParam("fileName") MultipartFile file, @RequestParam("minneighbors") int minneighbors, Map<String, Object> map){
        log.info("file [{}], size [{}], minneighbors [{}]", file.getOriginalFilename(), file.getSize(), minneighbors);

        String originalFileName = file.getOriginalFilename();
        if (!upload(file, uploadPath, originalFileName)){
            map.put("msg", "上传失败！");
            return "forward:/index";
        }

        String realPath = uploadPath + "/" + originalFileName;

        Mat srcImg = Imgcodecs.imread(realPath);

        // 目标灰色图像
        Mat dstGrayImg = new Mat();
        // 转换灰色
        Imgproc.cvtColor(srcImg, dstGrayImg, Imgproc.COLOR_BGR2GRAY);
        // OpenCv人脸识别分类器
        CascadeClassifier classifier = new CascadeClassifier(modelPath);
        // 用来存放人脸矩形
        MatOfRect faceRect = new MatOfRect();

        // 特征检测点的最小尺寸
        Size minSize = new Size(32, 32);
        // 图像缩放比例,可以理解为相机的X倍镜
        double scaleFactor = 1.2;
        // 执行人脸检测
        classifier.detectMultiScale(dstGrayImg, faceRect, scaleFactor, minneighbors, CV_HAAR_DO_CANNY_PRUNING, minSize);
        //遍历矩形,画到原图上面
        // 定义绘制颜色
        Scalar color = new Scalar(0, 0, 255);

        Rect[] rects = faceRect.toArray();

        // 没检测到
        if (null==rects || rects.length<1) {
            // 显示图片
            map.put("msg", "未检测到人脸");
            // 文件名
            map.put("fileName", originalFileName);

            return "forward:/index";
        }

        // 逐个处理
        for(Rect rect: rects) {
            int x = rect.x;
            int y = rect.y;
            int w = rect.width;
            int h = rect.height;
            // 单独框出每一张人脸
            Imgproc.rectangle(srcImg, new Point(x, y), new Point(x + w, y + w), color, 2);
        }

        // 添加人脸框之后的图片的名字
        String newFileName = UUID.randomUUID().toString() + ".png";

        // 保存
        Imgcodecs.imwrite(uploadPath + "/" + newFileName, srcImg);

        // 显示图片
        map.put("msg", "一共检测到" + rects.length + "个人脸");
        // 文件名
        map.put("fileName", newFileName);

        return "forward:/index";
    }
    /**
     * 显示单张图片
     * @return
     */
    @RequestMapping("show")
    public ResponseEntity showPhotos(String fileName){
        if (null==fileName) {
            return ResponseEntity.notFound().build();
        }

        try {
            // 由于是读取本机的文件，file是一定要加上的， path是在application配置文件中的路径
            return ResponseEntity.ok(resourceLoader.getResource("file:" + uploadPath + "/" + fileName));
        } catch (Exception e) {
            return ResponseEntity.notFound().build();
        }
    }
}

• The code of <font color="blue">UploadController.java</font> has the following points to pay attention to:

1. In the static method, use <font color="blue">System.loadLibrary</font> to load the local library. In the actual development process, this is the easiest place to report errors. Make sure that <font color="red">- Djava.library.path </ font> native library path configuration parameters are normally available, hereinbefore base image has been prepared quasi better than those of native libraries, just make sure <font color = "red"> -Djava.library.path</font> parameter configuration is correct, this configuration will be mentioned in the Dockerfile later
2. The <font color="blue">public String upload</font> method is the code entry for processing face detection, which is executed internally according to the sequence of the previous analysis process
3. <font color="blue">new CascadeClassifier(modelPath)</font> is to instantiate the classifier according to the specified model. The model file is downloaded from GitHub. Opencv officially trained the model in advance. The address is: https: //github.com/opencv/opencv/tree/master/data/haarcascades
4. The seemingly magical face detection function actually only needs one line of code <font color="blue">classifier.detectMultiScale</font> to get the rectangular position of each face in the original image. Next, we just need to follow Add a rectangular box to the original image

• Now that the code has been written, it will be made into a docker image

Docker image production

• The first is to write a Dockerfile:

# 基础镜像集成了openjdk8和opencv3.4.3
FROM bolingcavalry/opencv3.4.3:0.0.3

# 创建目录
RUN mkdir -p /app/images && mkdir -p /app/model

# 指定镜像的内容的来源位置
ARG DEPENDENCY=target/dependency

# 复制内容到镜像
COPY ${DEPENDENCY}/BOOT-INF/lib /app/lib
COPY ${DEPENDENCY}/META-INF /app/META-INF
COPY ${DEPENDENCY}/BOOT-INF/classes /app

# 指定启动命令
ENTRYPOINT ["java","-Djava.library.path=/opencv-3.4.3/build/lib","-cp","app:app/lib/*","com.bolingcavalry.facedetect.FaceDetectApplication"]

• The content of the above Dockerfile is very simple. It is the processing of copying files. There is only one thing to pay special attention to: there is a parameter in the startup command <font color="blue">-Djava.library.path=/opencv-3.4.3/build/ lib</font>, specifies the location of the local so library. In the previous java code, the local library loaded by <font color="blue">System.loadLibrary</font> is loaded from this location, the basis for our use The mirror image is <font color="blue">bolingcavalry/opencv3.4.3:0.0.3</font>, and all the local libraries of opencv have been prepared at this location
• Execute <font color="blue">mvn clean package -U</font> in the parent project directory. This is a pure maven operation and has nothing to do with docker
• Enter the <font color="blue">face-detect-demo</font> directory and execute the following command. The function is to extract the contents of class, configuration files, dependent libraries, etc. from the jar file to the target/dependency directory:

mkdir -p target/dependency && (cd target/dependency; jar -xf ../*.jar)

• Finally, execute the command <font color="blue">docker build -t bolingcavalry/facedetect:0.0.1 .</font> in the directory where the Dockerfile file is located (there is a point at the end of the command, don’t miss it) to complete the mirroring Make
• If you have an account with hub.docker.com, you can also push the image to the central warehouse through the docker push command, so that more people can use:
• Finally, let’s review "Three-Minute Speed Experience: Java Edition Face Detection" , as can be seen below, through the two -v parameters, the host's directory is mapped to the container. Therefore, the container The /app/images and /app/model in the can remain unchanged, as long as the directory mapping of the host can be guaranteed to be correct:

docker run \
--rm \
-p 18080:8080 \
-v /root/temp/202107/17/images:/app/images \
-v /root/temp/202107/17/model:/app/model \
bolingcavalry/facedetect:0.0.1

• For more information about the production of docker images officially recommended by SpringBoot, please refer to "SpringBoot (2.4) application production Docker image (Gradle version official plan)"

Need to pay attention to

• Please pay attention to the versions of several libraries related to javacv in pom.xml. These versions cannot be matched casually. It is recommended to follow the article. Even if you want to change, please check whether the version you need exists in the maven central warehouse;
• At this point, "Java Edition Face Detection" has been completed from experience to detailed development. The small functions involve a lot of knowledge points. It also allows us to experience the convenience and power of javacv. With the help of docker, the environment configuration and application development are separated. Come, reduce the difficulty of application development and deployment (no longer spend time on the deployment of jdk and opencv), if you are looking for a simple and easy-to-use javacv development and deployment solution, I hope this article can provide you with a reference;

You are not alone, Xinchen and original are with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. database + middleware series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos