Awesome! Next-generation Docker image building artifact

Docker automatically builds the image by reading the instructions in the Dockerfile, which is a text file that in turn contains all the commands needed to build a given image.

The above explanation is taken from Docker's official documentation and summarizes the purpose of Dockerfile. The use of Dockerfile is very important because it is our blueprint and a record of the layers we added to the Docker image.

In this article, we will learn how to use BuildKit features, which is a set of enhancements introduced on Docker v18.09. Integrating BuildKit will provide us with better performance, storage management and security.

prerequisites

Docker conceptual knowledge
Docker installed (currently using v19.03)
A Java application (in this article, I used a Jenkins Maven sample application)

let's start!

Simple Dockerfile example

The following is an example of an unoptimized Dockerfile containing a Java application. We will gradually make some optimizations.

FROM debian
COPY . /app
RUN apt-get update
RUN apt-get -y install openjdk-11-jdk ssh emacs
CMD [“java”, “-jar”, “/app/target/my-app-1.0-SNAPSHOT.jar”]

Here, we might ask ourselves: How long does it take to build? To answer this question, let's create the Dockerfile on the local development environment and let Docker build the image.

# enter your Java app folder
cd simple-java-maven-app-master
# create a Dockerfile
vim Dockerfile
# write content, save and exit
docker pull debian:latest # pull the source image
time docker build --no-cache -t docker-class . # overwrite previous layers
# notice the build time
0,21s user 0,23s system 0% cpu 1:55,17 total

At this time, our build requires 1m55s.

If we only enable BuildKit without other changes, will it be different?

Enable BuildKit

BuildKit can be enabled in two ways:

Set the DOCKER_BUILDKIT = 1 environment variable when calling the Docker build command, for example:

time DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class

To set Docker BuildKit to be enabled by default, you need to set the following settings in /etc/docker/daemon.json, and then restart:

{ "features": { "buildkit": true } }

The initial effect of BuildKit

DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class .
0,54s user 0,93s system 1% cpu 1:43,00 total

At this time, our build requires 1m43s. On the same hardware, the time it takes to build is about 12 seconds less than before. This means that the construction can save about 10% of the time with almost no effort.

Let us now see if we can take some additional steps to further improve.

Order from smallest to most frequent

Because order is important for caching, we moved the COPY command closer to the end of the Dockerfile.

FROM debian
RUN apt-get update
RUN apt-get -y install openjdk-11-jdk ssh emacs
RUN COPY . /app
CMD [“java”, “-jar”, “/app/target/my-app-1.0-SNAPSHOT.jar”]

Avoid using "COPY."

Choose more specific COPY parameters to avoid cache interruption. Copy only what you need.

FROM debian
RUN apt-get update
RUN apt-get -y install openjdk-11-jdk ssh vim
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

apt-get update and install command used together

This prevents the use of outdated package caches.

FROM debian
RUN apt-get update && \
    apt-get -y install openjdk-11-jdk ssh vim
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Remove unnecessary dependencies

In the beginning, do not install debugging and editing tools, you can install them later when needed.

FROM debian
RUN apt-get update && \
    apt-get -y install --no-install-recommends \
    openjdk-11-jdk
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Delete the package manager cache

Your mirror does not need this cached data. Take this opportunity to free up some space.

FROM debian
RUN apt-get update && \
    apt-get -y install --no-install-recommends \
    openjdk-11-jdk && \
    rm -rf /var/lib/apt/lists/*
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Use official mirrors whenever possible

There are many reasons to use official images, such as reducing image maintenance time and reducing image size, and pre-configuring images for container use.

FROM openjdk
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Use specific tags

Do not use the latest tag.

FROM openjdk:8
COPY target/my-app-1.0-SNAPSHOT.jar /app
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Find the smallest mirror

The following is a list of openjdk mirrors. Choose the lightest mirror that suits you best.

REPOSITORY TAG标签 SIZE大小
openjdk 8 634MB
openjdk 8-jre 443MB
openjdk 8-jre-slim 204MB
openjdk 8-jre-alpine 83MB

Build from source in a consistent environment

If you don't need the entire JDK, you can use the Maven Docker image as a build basis.

FROM maven:3.6-jdk-8-alpine
WORKDIR /app
COPY pom.xml .
COPY src ./src
RUN mvn -e -B package
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Get dependencies in a separate step

Can be cached-Dockerfile command used to get dependencies. Caching this step will speed up the build.

FROM maven:3.6-jdk-8-alpine
WORKDIR /app
COPY pom.xml .
RUN mvn -e -B dependency:resolve
COPY src ./src
RUN mvn -e -B package
CMD [“java”, “-jar”, “/app/my-app-1.0-SNAPSHOT.jar”]

Multi-stage build: remove build dependencies

Why use a multi-stage build?

Separate the build from the runtime environment
DRY way
- With different detailed information of development, testing and other environments
Linearization dependency

Has platform-specific stages

FROM maven:3.6-jdk-8-alpine AS builder
WORKDIR /app
COPY pom.xml .
RUN mvn -e -B dependency:resolve
COPY src ./src
RUN mvn -e -B package

FROM openjdk:8-jre-alpine
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

If you build our application at this time,

time DOCKER_BUILDKIT=1 docker build --no-cache -t docker-class .
0,41s user 0,54s system 2% cpu 35,656 total

You will notice that our application takes about 35.66 seconds to build. This is a pleasant improvement.

Next, we will introduce the functions of other scenes.

Multi-stage construction: different mirroring styles

The following Dockerfile shows the different stages of Debian-based and Alpine-based mirroring.

FROM maven:3.6-jdk-8-alpine AS builder
…
FROM openjdk:8-jre-jessie AS release-jessie
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

FROM openjdk:8-jre-alpine AS release-alpine
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

To build a specific image, we can use the –target parameter:

time docker build --no-cache --target release-jessie .

Different mirroring styles (DRY/Global ARG)

ARG flavor=alpine
FROM maven:3.6-jdk-8-alpine AS builder
…
FROM openjdk:8-jre-$flavor AS release
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

The ARG command can specify the image to be built. In the above example, we specified alpine as the default image, but we can also specify the image through the --build-arg flavor= parameter in the docker build command.

time docker build --no-cache --target release --build-arg flavor=jessie .

Concurrent

Concurrency is important when building a Docker image because it makes full use of the available CPU threads. In a linear Dockerfile, all stages are executed sequentially. With a multi-phase build, we can prepare the smaller dependent phases for the main phase to use them.

BuildKit even brings another performance benefit. If this stage is not used in future builds, these stages will be skipped directly at the end instead of being processed and discarded.

Here is an example Dockerfile, where the assets of the website are built in an assets phase:

FROM maven:3.6-jdk-8-alpine AS builder
…
FROM tiborvass/whalesay AS assets
RUN whalesay “Hello DockerCon!” > out/assets.html

FROM openjdk:8-jre-alpine AS release
COPY --from=builder /app/my-app-1.0-SNAPSHOT.jar /
COPY --from=assets /out /assets
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

This is another Dockerfile in which the C and C++ libraries are compiled separately, and this stage is used after the builder.

FROM maven:3.6-jdk-8-alpine AS builder-base
…

FROM gcc:8-alpine AS builder-someClib
…
RUN git clone … ./configure --prefix=/out && make && make install

FROM g++:8-alpine AS builder-some CPPlib
…
RUN git clone … && cmake …

FROM builder-base AS builder
COPY --from=builder-someClib /out /
COPY --from=builder-someCpplib /out /

BuildKit application cache

BuildKit has a special function of package manager caching. Here are some examples of cache folder locations:

Package manager path

apt /var/lib/apt/lists
go ~/.cache/go-build
go-modules $GOPATH/pkg/mod
npm ~/.npm
pip ~/.cache/pip

We can compare this Dockerfile with the Dockerfile described above in Building from Source in a Consistent Environment. This earlier Dockerfile has no special cache handling. We can use --mount=type=cache to do this.

FROM maven:3.6-jdk-8-alpine AS builder
WORKDIR /app
RUN --mount=target=. --mount=type=cache,target /root/.m2 \
    && mvn package -DoutputDirectory=/

FROM openjdk:8-jre-alpine
COPY --from=builder /app/target/my-app-1.0-SNAPSHOT.jar /
CMD [“java”, “-jar”, “/my-app-1.0-SNAPSHOT.jar”]

Security features of BuildKit

BuildKit has security features. In the following example, we use --mount=type=secret to hide some confidential files, such as ~/.aws/credentials.

FROM <baseimage>
RUN …
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials,required \
./fetch-assets-from-s3.sh
RUN ./build-scripts.sh

To build this Dockerfile, you need to use the –secret parameter:

docker build --secret id=aws,src=~/.aws/credentials

In order to improve security and avoid using commands such as COPY ./keys/private.pem /root .ssh/private.pem, we can use ssh in BuildKit to solve this problem:

FROM alpine
RUN apk add --no-cache openssh-client
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
ARG REPO_REF=19ba7bcd9976ef8a9bd086187df19ba7bcd997f2
RUN --mount=type=ssh,required git clone git@github.com:org/repo /work && cd /work && git checkout -b $REPO_REF

To build this Dockerfile, you need to load your SSH private key in ssh-agent.

eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa # this is the SSH key default location
docker build --ssh=default .

in conclusion

In this article, we introduced the use of Docker BuildKit to optimize Dockerfile, and therefore speed up the image build time. These speed improvements can help us improve efficiency and save computing power.

Source: https://os.51cto.com/art/202104/660131.htm