Container Image Security in the Cloud Native Era (Part 1)

Hi everyone, this is Zhang Jintao.

As the cornerstone of cloud native, Kubernetes has brought us great convenience. More and more companies are also applying Kubernetes to the production environment. However, while enjoying the convenience it brings, we also need to pay attention to some of the potential safety hazards.

In this article, I will focus on the content related to Container Image Security

Under normal circumstances, we mentioned container image security , mainly referring to the following two aspects:

• The security of the content of the mirror itself;
• Security of the image distribution process;

Mirror the security of its own content

To talk about the security of the content of the mirror itself, then we need to know what the mirror is and what its content is.

What is mirror

Let's take the debian image as an example, pull the latest image, save it as a tar file, and then decompress it:

➜  ~ mkdir -p debian-image
➜  ~ docker pull debian
Using default tag: latest
latest: Pulling from library/debian
647acf3d48c2: Pull complete 
Digest: sha256:e8c184b56a94db0947a9d51ec68f42ef5584442f20547fa3bd8cbd00203b2e7a
Status: Downloaded newer image for debian:latest
docker.io/library/debian:latest
➜  ~ docker image save -o debian-image/debian.tar debian
➜  ~ ls debian-image 
debian.tar
➜  ~ tar -C debian-image -xf debian-image/debian.tar 
➜  ~ tree -I debian.tar debian-image 
debian-image
├── 827e5611389abf13dad1057e92f163b771febc0bcdb19fa2d634a7eb0641e0cc.json
├── b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda
│   ├── json
│   ├── layer.tar
│   └── VERSION
├── manifest.json
└── repositories

1 directory, 6 files

After the decompression is complete, we see that it is a layer.tar files, and we decompress layer.tar

➜  ~ tar -C debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda -xf debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda/layer.tar
➜  ~ tree -I 'layer.tar|json|VERSION'  -L 1 debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda
debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda
├── bin
├── boot
├── dev
├── etc
├── home
├── lib
├── lib64
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin
├── srv
├── sys
├── tmp
├── usr
└── var

19 directories, 0 files

You must be familiar with the unzipped directory structure, yes, this is the directory structure rootfs

If we are using some application images built by ourselves, after several decompressions, you will find the files corresponding to the application in them.

How to ensure the security of the content of the mirror itself

Earlier we have seen that the container image is a rootfs and the application, as well as some configuration files. Therefore, to ensure the security of its own content, it is mainly considered from the following aspects:

rootfs security

Corresponds to our actual situation, rootfs is usually provided by a base (system) we use the mirror, or may be considered as we build mirror Dockerfile the FROM configured to provide the image field.

In this respect, we need to be safe:

• Use mirrors from trusted sources, such as those officially maintained by Docker;
• Continuously scan and upgrade the basic image for vulnerabilities;
• You can also consider using the Distroless mirror, which can also be protected from attacks to a certain extent;

application

The application program is actually provided by ourselves. If we want to be safe in this respect, we need to:

• Continuously scan the software for vulnerabilities;
• Update dependencies in a timely manner;
• Consider transitioning from SDL (Security Development Lifecycle) to DevSecOps;

Configuration file

The configuration files contained in the image are provided by the image building tool. Generally, as long as we ensure that the image building tool used has not been tampered with or left any loopholes, there will basically be no major problems here.

Taken together, we can directly use Trivy or Anchore Engine to help us protect the security of mirrored content. In addition, some mirror warehouses, such as Harbor, have built-in mirror security scanning tools, or you can use the docker scan command to perform mirror security scanning.

Image distribution security

How to distribute the image

Let's first take a look at how the container image is built to deployed to our Kubernetes environment.

Figure 1, the brief process diagram of the container image from creation to release deployment

After the developer writes the code, he pushes the code to the code repository. This triggers CI to build, during which the image will be built and the image will be pushed to the image warehouse.

In the CD link, the image in the mirror warehouse will be used to deploy to the target Kubernetes cluster.

So in this process, how does an attacker conduct an attack?

Security issues in image distribution

Figure 2, Image distribution deployment security example

As shown in the figure, in the image distribution and deployment link, the upstream is the mirror warehouse, and the downstream is the Kubernetes cluster. For mirrored warehouses, even in the self-built environment of the intranet, since our concept has changed from border-based security to zero-trust security, we unified the public warehouse as an example to explain.

Attackers can hijack and replace them with malicious images by some means, including directly attacking image warehouses.

To ensure the security source and integrity of the image deployed to the Kubernetes cluster, it actually needs to be carried out in two main links:

• image when building the image;
• Perform signature verification during image distribution and deployment; (continue to the next article)

Let's take a look at them separately.

Mirrored label and summary

We usually have two choices when using container images:

• Label, such as alpine:3.14.3
• Abstract, such as alpine@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2

In most scenarios, we will use the label directly because it is more readable. However, the content of the mirror image may change over time, because we may use the same label for the mirror image of different content, the most common is the :latest , every time a new version is released, the new version of the mirror will continue to use :latest Label, but the application version in it has been upgraded to the latest.

The main disadvantage of using the digest is its poor readability. However, the digest of each image is unique. The digest is the SHA256 hash value of the content of the image. So we can guarantee the uniqueness of the image through the summary.

You can directly see the label and summary information through the following example:

➜  ~ docker pull alpine:3.14.3                                                                          
3.14.3: Pulling from library/alpine
Digest: sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2
Status: Image is up to date for alpine:3.14.3
docker.io/library/alpine:3.14.3
➜  ~ docker image inspect alpine:3.14.3 | jq -r '.[] | {RepoTags: .RepoTags, RepoDigests: .RepoDigests}'
{
  "RepoTags": [
    "alpine:3.14.3"
  ],
  "RepoDigests": [
    "alpine@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2"
  ]
}

So how to ensure the correctness/safety of the mirroring? This is the main problem that the mirror signature solves.

Mirror signature solution

Digital signature is a well-known method used to maintain the integrity of any data transmitted on the network. For container image signing, we have several common schemes.

Docker Content Trust (DCT)

When transferring general files, you may have similar experiences, such as incomplete files downloaded due to network reasons; or encountering man-in-the-middle attacks, causing files to be tampered with or replaced.

In fact, mirroring may encounter similar problems during the distribution process. This is the focus of our discussion here, which is the main problem that Docker Content Trust (DCT) solves.

Docker Content Trust uses digital signatures and allows the client or runtime to verify the integrity and publisher of a specific image tag. For use, it is the related functions provided by the docker trust Note: This requires Docker CE 17.12 and above.

As we mentioned earlier, the mirror record can have some tags, the format is as follows:

[REGISTRY_HOST[:REGISTRY_PORT]/]REPOSITORY[:TAG]

Take the label as an example, DCT will be associated with a part of the label. Each mirror warehouse has a set of keys, and the mirror publisher uses these keys to sign the mirror label. (The image publisher can decide which labels to sign.) The image repository can contain multiple images with signed and unsigned labels at the same time.

It needs to be explained here that if the image publisher first pushes the signed latest image and then pushes the unsigned latest image, then the latter image will not affect the content of the previous image (different from the place covered by the label above).

Figure 4, DCT mirror image signature example (the figure shows a simplified authentication process for logging in to the mirror warehouse)

In production, we can enable DCT to ensure that the images used are signed. If DCT is enabled, only trusted images (images that have been signed and verified) can be pulled, run, or built.

Enabling DCT is a bit like applying a "filter" to a mirror repository, that is, you can only see signed mirror tags, but not unsigned mirror tags. If the client does not enable DCT, then it can see all the mirrors.

Here we take a quick look at the working process of DCT

Its trust in the mirror label is managed through the use of a signature key. The key set will be created when we turn on DCT for the first time and use it. A key set consists of the following types of keys:

• An offline key, which is the root of the mirror label DCT ( is difficult to recover root key is lost 161a9883963311)
• The repository or tag key that signs the tag
• Server-managed keys, such as timestamp keys

Figure 5, the image signature key example

We just mentioned that the client uses DCT, which is our docker trust command, which is built on Notary v1. By default, DCT is disabled in the Docker client. To enable it, you need to set the DOCKER_CONTENT_TRUST=1 environment variable.

The effect is as follows:

➜  ~ DOCKER_CONTENT_TRUST=1 docker pull alpine:3.12
Pull (1 of 1): alpine:3.12@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
docker.io/library/alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a: Pulling from library/alpine
188c0c94c7c5: Already exists        
Digest: sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
Status: Downloaded newer image for alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
Tagging alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a as alpine:3.12 
docker.io/library/alpine:3.12

Notary v1

Earlier we mentioned that DCT is implemented based on Notary v1, but this is not the focus of our article, so here is only a brief introduction to Notary v1. Notary project address: https://github.com/notaryproject/notary

Figure 6, Notary client, server and signature-related interaction process

Process 1-Identity authentication, any connection without a token will be redirected to the authorization server (Docker Registry v2 identity authentication);

Process 2-The client will log in to the authorization server through HTTPS authentication and obtain the token;

Process 3-When the client uploads a new metadata file, the server will check whether there is any conflict between them according to the previous version, and verify the signature, checksum and validity of the uploaded metadata;

Process 4-Once all uploaded metadata has been verified, the server will generate a timestamp (and possibly a snapshot), and then send them to sign for signing;

Process 5-sign retrieves the encrypted private key from its database, decrypts the key, uses them to sign, and sends it back to the server;

Process 6-The server stores the metadata uploaded by the client and generated by the server in the TUF library. The generated time stamp and snapshot metadata prove that the metadata uploaded by the client is the latest version of the trusted set. After that, the server will notify the client that the upload was successful;

Process 7-The client can now download the latest metadata from the server immediately. In the event that the timestamp expires, the server will traverse the entire sequence, generate a new timestamp, request a sign, and store the timestamp of the new signature in the database. Then, it sends this new timestamp along with other stored metadata to the requesting client;

Because this project is a safety project, although it is very useful, it is not active as a whole. The v2 version is currently being developed, and interested friends are welcome to join.

sigstore and Cosign

Here is another project from the Linux Foundation called sigstore. It is mainly to provide some standard libraries/tools for better signing and verification. Of course, sigstore has gathered open source projects including Cosign, Fulcio and Rekor, which involve mirror image signature verification and supply chain.

Figure 7, sigstore introduction

Cosign is one of sigstore's tools for creating, storing and verifying container image signatures in the OCI registry. Cosign v1.0 was released in the second half of this year. Whether it can be used in a production environment is still to be tested. Up to now, Cosign has released the v1.3.1 version. For detailed changes, please refer to its ReleaseNote: https://github.com/sigstore/cosign/releases/tag/v1.3.1

Let's take a look at how it signs the image

➜  cosign cosign generate-key-pair                                                                                              
Enter password for private key: 
Enter password for private key again: 
Private key written to cosign.key
Public key written to cosign.pub
➜  cosign cosign sign --key cosign.key ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo:fa5714f419b3d11dee6ac795e38356e9c3c439cb
Enter password for private key: %   
➜  cosign cosign verify --key cosign.pub  ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo:fa5714f419b3d11dee6ac795e38356e9c3c439cb 

Verification for ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo:fa5714f419b3d11dee6ac795e38356e9c3c439cb --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo"},"image":{"docker-manifest-digest":"sha256:768845efa2a32bc5c5d83a6f7ec668b98f5db46585dd1918afc9695a9e653d2d"},"type":"cosign container image signature"},"optional":null}]

It seems to be relatively simple.

Summarize

The above is about the content security of the image itself, and the content of the image signature verification part of the image distribution security.

In the next article, I will introduce how to verify signatures during image distribution and deployment, and how to protect Kubernetes clusters from unsigned or untrusted source images. Stay tuned!

---

Welcome to subscribe to my article public account【MoeLove】