OCI and container image construction

Hi everyone, this is Zhang Jintao.

In this article, I will introduce OCI and Docker mirror related content, welcome to leave a message to discuss.

OCI's past and present

In March 2013, dotCloud Company conducted the first demonstration of Docker at PyCon, and subsequently announced the open source. Since then, Docker has been known to everyone, and then there has been a wave of containerization.

In June 2014, Docker 1.0 was officially released, with nearly 460 contributors and more than 8,700 submissions, which also marked that Docker has reached a state of production availability.

At the time, the first idea of containerization was to use Docker. At that time, the realization or development direction of Docker was mainly controlled by Docker Inc., and there was no uniform industry standard. This is obviously unacceptable for some head companies. The absence of a unified industry standard means that if you choose to use Docker’s containerization technology, you will be bound by Docker Inc.; Upgrade, some functions or features will inevitably change, no one can guarantee that there will be no destructive changes.

Therefore, in order to promote the industrial standardization of containerization technology, the Linux Foundation and Google, Huawei, HP, IBM, Docker, Red Hat, VMware and other companies jointly announced the establishment of the Open Container Project (OCP) at DockerCon in June 2015, which was later renamed For OCI. Its main goal is establish the container format and runtime industry open general standard .

Since its development, there are three main standards formulated by OCI: runtime-spec , image-spec and distribution-spec These three standards respectively define the container runtime, container image and distribution specifications, which will be introduced later.

In order to support the advancement of the OCI container runtime standard, Docker drafted a draft of the image format and runtime specification, and donated the relevant implementation of the Docker project to OCI as the basic implementation of the container runtime. The current project name is runc .

Later, Docker independently formed its container runtime into a project named containerd and donated this project to CNCF, which is now a CNCF graduated project.

OCI image vs Docker image

The establishment of OCI promoted the industrial standardization of container technology, but is this standard the only one? actually not. When the OCI was established and the image-spec standard was formulated, Docker had prospered unprecedentedly and was widely used.

Since the standard only defines the most basic content, if you want to transform all the implementation of Docker in accordance with the standard, it will cause destructive changes to Docker and is not conducive to the iteration of Docker functions.

Therefore, in order to support the popularization of the OCI standard, Docker has promoted the registry’s support for OCI images, and is now adding adaptations to Docker itself. The goal is to allow Docker to support two image formats, one that conforms to the Docker standard and one that conforms to the Docker standard. A mirror image of the OCI standard.

What are the similarities and differences between the two? Let's take a look at it step by step.

The difference and connection between Docker Image and OCI Image

In my previous article, we have introduced in detail what Docker image is, and here we will quickly introduce it.

Each Docker image is organized by a series of configuration checklists and corresponding layers. Each layer is generally archived in tar format. The configuration list describes the order in which the corresponding layers should be organized, and some meta attributes of the image. For example, the architecture supported by the mirror, such as amd64 , and some parameters configured in advance such as ENV.

Of course, Docker Image also contains information such as the Docker version docker_version used when building the image and the history of building the image history . So you can see the Docker version used to build the image DockerHub or other mirror repositories, or you can view the build history docker history <IMAGE>

So what is OCI Image? First of all, we need to have an OCI Image to explore what it is.

Here is a tool skopeo can be easily converted from a mirror warehouse or a local Docker daemon or even a Docker Image tar file saved docker save

skopeo will not be repeated here, just refer to the documentation on the project homepage. Start using it directly here.

We use the mirror image of debian

(MoeLove) ➜  skopeo copy docker://debian:stretch oci:debian:stretch    
Getting image source signatures
Copying blob a4d8138d0f6b done
Copying config 45f82268e3 done
Writing manifest to image destination
Storing signatures

You will get an OCI Image through the above command. Let's take a look at its directory structure.

(MoeLove) ➜  tree debian 
debian
├── blobs
│   └── sha256
│       ├── 0043cd2a654fe86258f43f5b1dbbb4e6c582cc4bb6e505e9c5171c124150d155
│       ├── 45f82268e32180cb1839f90467d9b8a8258953d68b7221199976653308d92ef5
│       └── a4d8138d0f6b5a441aaa533faf5fe0c3996a6ca42643c46f4402c7e8bda53742
├── index.json
└── oci-layout

2 directories, 5 files

Is there a feeling of deja vu? Yes, the OCI Image specification is based on Docker Image, so it seems that the difference is not particularly large. Let's take a look at the specific content.

oci-layout

This file is the layout file of OCI Image, and it is also used to explain the mirroring specifications it uses or follows.

(MoeLove) ➜  debian cat oci-layout| jq
{
  "imageLayoutVersion": "1.0.0"
}

You can see that the content here is written as 1.0.0 which means that the image follows the layout specification of OCI version 1.0.0.

index.json

index.json file manifest field similar to the Docker Image manifest.json as OCI Image top configuration, is a mirror image of the configuration entry.

(MoeLove) ➜  debian cat index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:0043cd2a654fe86258f43f5b1dbbb4e6c582cc4bb6e505e9c5171c124150d155",
      "size": 349,
      "annotations": {
        "org.opencontainers.image.ref.name": "stretch"
      },
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    }
  ]
}

From its content, it can be seen that the mediaType field is the same as the type in Docker Image, but the docker has been replaced by oci . From this configuration file, we can find that the first blob is sha256:0043cd2a654fe86258f43f5b1dbbb4e6c582cc4bb6e505e9c5171c124150d155 Let's take a look at its content.

(MoeLove) ➜  debian cat blobs/sha256/0043cd2a654fe86258f43f5b1dbbb4e6c582cc4bb6e505e9c5171c124150d155 | jq
{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:45f82268e32180cb1839f90467d9b8a8258953d68b7221199976653308d92ef5",
    "size": 579
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:a4d8138d0f6b5a441aaa533faf5fe0c3996a6ca42643c46f4402c7e8bda53742",
      "size": 45337510
    }
  ]
}

This entry file describes the actual configuration of the OCI image and the layer configuration in it. If there are multiple layers, layers will increase accordingly.

Note: mediaType used in application/vnd.oci.image.layer.v1.tar+gzip means that the data content is compressed by gzip if you are interested, you can decompress it with tar, you will find very interesting content.

Let me tell you the result first. After decompression, you will get a rootfs which is similar to Docker Image.

summary

We used skopeo tool to obtain the OCI Image from the debian Docker Image from the local Docker daemon, and analyzed its contents.

The main difference is that their directory structure is not exactly the same, and the configuration information, especially the specification of mediaType , is different.

And their relationship lies in this. The OCI Image specification is modified from the Docker Image specification, so the organization of blobs similar to them is roughly the same, and many parameters in the configuration file are similar.

In addition, we can easily get another conclusion, that is, we can easily convert Docker Image to OCI Image.

Conversion of OCI image and Docker image

As we have seen above, using skopeo tool can convert Docker Image to OCI Image, of course it can also convert OCI Image to Docker Image. The method is given below:

# 从 DockerHub 将 debian 的 Docker Image 拉取并转换为 OCI Image
(MoeLove) ➜  skopeo copy docker://debian:stretch oci:debian:stretch    
Getting image source signatures
Copying blob a4d8138d0f6b done
Copying config 45f82268e3 done
Writing manifest to image destination
Storing signatures


# 将当前目录下的 debian 的 OCI Image 转换为 Docker Image 并存储到本地 docker daemon 中
(MoeLove) ➜  skopeo copy  oci:debian:stretch docker-daemon:local/debian:oci
Getting image source signatures
Copying blob 0e350e141713 done
Copying config aae58a37cf done
Writing manifest to image destination
Storing signatures
# 验证
(MoeLove) ➜  oci docker images local/debian
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
local/debian        oci                 ac6bcf605d82        6 months ago        101MB

Image build tool

In the CI/CD environment, although we can use DinD (Docker in Docker) to start a docker daemon or use the mount method to mount the external /var/run/docker.sock into the container, or expose the Docker API using HTTP Come out, directly use the address to build.

But do you think these methods are heavier? Have you considered safety issues, or pressure and load issues?

The pressure and load here mainly refer to the pressure on the docker daemon when all tasks share the same docker daemon to provide services.

Here we introduce some other image building tools, using these tools allows you to build an image in a Docker-free environment and upload it to the Docker image warehouse.

So far, we have many choices:

• BuildKit
• img
• orca-build
• umoci
• buildah
• kaniko
• FTL
• Bazel rules_docker

These tools have different focuses. Of course, these tools are not the only ones listed above, but these tools are more typical.

buildah as a next-generation image building tool on the Internet. The main reason is that it can directly build OCI standard images or Docker images, or you can directly use Dockerfile . And it can also pull / push . It can be said that it is fully compatible with Docker in terms of image construction. It can even be said that it can be used as a substitute for Docker in terms of image construction.

And buildah does not need any root permissions when building the image, and does not rely on Docker. It uses a simple fork-exec model, and it can also be included as a library in other tools. Its ultimate goal is to provide a lower-level core tool set to accomplish things related to building mirrors.

After talking about this typical alternative, let's BuildKit and img . img is built on BuildKit , so there are many similarities. They use non-root users to build images. Of course, BuildKit I introduced in detail in the previous article, it is the next generation build tool built in Docker, and it can be used independently. It is not an exaggeration to call it the "next generation image building tool".

kaniko was launched by Google. Its main slogan is "Container Mirroring in Kubernetes". In fact, it can work whether in a K8S cluster or in a container. It can also use Dockerfile build an image. Of course, there is a very important point. All its build commands are run in user mode, and can also be well integrated with Kubernetes. In the cloud native era, it also occupies a certain advantage.

The above tools are just a general introduction. If you are interested in them, you can go directly to the project homepage to view the README.md . There are more detailed instructions, so I won't repeat them here.

Summarize

This article introduces the past and present of OCI, as well as the specifications and characteristics of OCI Image, and also introduces a tool skopeo that can be used to convert between OCI Image and Docker Image. In addition, some tools that can be used to replace Docker build in a CI environment or other environments with specific scenarios are introduced. Please choose according to your actual needs.

---

Welcome to subscribe to my article public account【MoeLove】