Why is it so hard to install Docker on Apple Silicon

The picture shows the internal Wiki, we have tried various Docker development environments

Recently, many colleagues in the company have switched to the new MacBook Pro equipped with M1 Pro or M1 Max. Although the daily software such as Chrome, Visual Studio Code and Slack have been adapted well, it is difficult to face Docker. .

As we all know, Docker uses two features of Linux: namespaces and cgroups to provide isolation and resource constraints, so in any case on macOS we must use Docker through a virtual machine.

In April 2021, Docker for Mac (Docker Desktop) released experimental support for Apple Silicon, which will use QEMU to run an ARM-based Linux virtual machine, which runs ARM-based images by default, but also supports running Mirror for x86.

QEMU is an open source virtual machine (Virtualizer) and emulator (Emulator). The so-called emulator means that QEMU can simulate running a computer without virtualization support from hardware or operating system, including simulation A different CPU architecture from the host, such as a computer that emulates the x86 architecture on Apple Silicon. In the case of hardware virtualization support, QEMU can also use the host's CPU to run directly, reducing the performance overhead of simulation running, such as using Hypervisor.Framework provided by macOS.

Docker for Mac actually uses these two capabilities of QEMU to run x86 images on ARM virtual machines and run ARM virtual machines on Mac.

Docker for Mac is really good, in addition to solving the problems brought by the new architecture, it also maps the file system and the network, the container can access the file system or expose the network port to the machine as if it were running on the machine, it almost feels Without the presence of a virtual machine. But LeanCloud is no longer a small company after joining TapTap. According to the new price plan launched by Docker Desktop in August , each of us needs to pay a monthly subscription fee of at least $5. It's not that we don't want to pay this money, it's just that I want to find an open source solution.

Previously on Intel Macs, we would use Vagrant or minikube to create virtual machines, which would use VirtualBox or HyperKit under the hood for the actual virtualization. But neither VirtualBox nor HyperKit have plans to support Apple Silicon. In fact, among the current open source virtualization solutions, only QEMU has good support for Apple Silicon. QEMU itself only provides a command line interface. For example, the command line parameters when Docker for Mac calls QEMU are as follows:

/Applications/Docker.app/Contents/MacOS/qemu-system-aarch64 -accel hvf \
-cpu host -machine virt,highmem=off -m 2048 -smp 5 \
-kernel /Applications/Docker.app/Contents/Resources/linuxkit/kernel \
-append linuxkit.unified_cgroup_hierarchy=1 page_poison=1 vsyscall=emulate \
panic=1 nospec_store_bypass_disable noibrs noibpb no_stf_barrier mitigations=off \
vpnkit.connect=tcp+bootstrap+client://192.168.65.2:61473/f1c4db329a4a520d73a79eaa1360de7be7d09948a1ac348b04c8e01f6f6eb2c9 \
console=ttyAMA0 -initrd /Applications/Docker.app/Contents/Resources/linuxkit/initrd.img \
-serial pipe:/var/folders/12/_bbrd4692hv8r9bx_ggw5kp80000gn/T/qemu-console1367481183/fifo \
-drive if=none,file=/Users/ziting/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw,format=raw,id=hd0 \
-device virtio-blk-pci,drive=hd0,serial=dummyserial -netdev socket,id=net1,fd=3 -device virtio-net-device,netdev=net1,mac=02:50:00:00:00:01 \
-vga none -nographic -monitor none

In order to actually use QEMU for development, we need a more user-friendly package that automatically configures Docker and Kubernetes (or at least makes it easy to write scripts like Vagrantfile), provides Docker for Mac-like network mapping and file mapping, so I Found Lima.

Lima itself as a macOS subsystem for Linux. It uses QEMU to run a Linux virtual machine with containerd in rootless mode, and provides file mapping and automatic port forwarding via SSH.

But why containerd and not Docker? With the rise of the container orchestration platform Kubernetes, the community hopes to standardize the key link of running containers, making it easier to introduce other container runtimes other than Docker, so launched Container Runtime Interface (CRI). containerd is a CRI implementation split from Docker, which is more streamlined than Docker ontology, and is now maintained by the community.

Therefore, new open source software such as Lima will prefer containerd to run containers, because the components are more streamlined and have better performance, and they are not easily affected by changes at the Docker product level. nerdctl is a command-line client for containerd ( nerd is the last 4 letters of containerd ) and is used similar to (but not fully compatible with) docker or docker-compose.

The so-called rootless means that by replacing some components, the container runtime (containerd) and the container run under the non-root user, each user has its own containerd, so that most operations do not need to switch to root. It also reduces the attack surface of security breaches.

But we hope to run the complete rootful mode of dockerd and Kubernetes locally to simulate the real online environment as much as possible. Fortunately, Lima provides a wealth of customization capabilities , I am based on some scripts in the community ( docker. yaml and minikube.yaml ) meet our needs, and these custom logics are written in the yaml description file in the form of scripts, and the same virtual machine can be created with just one command.

~ ❯ limactl start docker.yaml
? Creating an instance "docker" Proceed with the default configuration
INFO[0005] Attempting to download the image from "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img"
INFO[0005] Using cache "/Users/ziting/Library/Caches/lima/download/by-url-sha256/ae20df823d41d1dd300f8866889804ab25fb8689c1a68da6b13dd60a8c5c9e35/data"
INFO[0006] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/ziting/.lima/docker/serial.log")
INFO[0006] SSH Local Port: 55942
INFO[0006] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0039] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0039] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0048] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf to contain \"user_allow_other\""
INFO[0051] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running"
INFO[0051] [hostagent] Mounting "/Users/ziting"
INFO[0051] [hostagent] Mounting "/tmp/lima"
INFO[0052] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/ziting/.lima/docker/ga.sock" (host)
INFO[0092] [hostagent] Waiting for the optional requirement 1 of 1: "user probe 1/1"
INFO[0154] [hostagent] Forwarding TCP from [::]:2376 to 127.0.0.1:2376
INFO[0304] [hostagent] Forwarding TCP from [::]:8443 to 127.0.0.1:8443
INFO[0332] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0351] READY. Run `limactl shell docker` to open the shell.
INFO[0351] To run `docker` on the host (assumes docker-cli is installed):
INFO[0351] $ export DOCKER_HOST=tcp://127.0.0.1:2376
INFO[0351] To run `kubectl` on the host (assumes kubernetes-cli is installed):
INFO[0351] $ mkdir -p .kube && limactl cp minikube:.kube/config .kube/config

I also found another Lima-based package - Colima , which provides rootful dockerd and Kubernetes by default, but Colima does not expose Lima's powerful customization capabilities, so we don't use it, but for development with less requirements It is also an easier option for users.

By default, Docker in Lima can only run ARM architecture images on Apple Silicon, but as mentioned earlier, we can use QEMU's emulation capabilities to run containers of other architectures (such as x86) . qemu-user-static is a process-level emulator that can run executable files of other architectures like an interpreter. We can use a feature of Linux Binfmt_misc ( Chinese version ) to allow Linux to encounter executables of specific architectures qemu-user-static is automatically called when the file is opened, this ability also applies to executable files in containers.

There are also projects such as qus in the community, which encapsulate these capabilities, and only need to execute a line docker run --rm --privileged aptman/qus -s -- -p x86_64 to make your ARM virtual machine magically support running x86 images.

/usr/bin/containerd-shim-runc-v2
 \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;
     \_ /qus/bin/qemu-x86_64-static /usr/sbin/nginx -g daemon off;

The process tree of running x86 image using qus is as above, all processes (including created subprocesses) are automatically run through QEMU simulation.

Back to the question in the title, because Docker relies on the characteristics of the Linux kernel, it must be run through a virtual machine on Mac; Apple Silicon is a new architecture, the choice of virtual machine is more limited, because some images do not provide ARM architecture Docker images, so sometimes there is a need to simulate running x86 images; Docker Desktop, as a commercial product, has enough energy to solve these "dirty jobs", but it chooses to no longer allow everyone to use it for free at this point in time; New projects in the open source community all hope to de-Dockerize and replace dockerd with containerd, but this brings changes in usage habits and may be inconsistent with the online environment. For these reasons, installing Docker on Apple Silicon currently requires some background knowledge, but there are still some great open source projects to choose from.

Although Cloud Engine is also built based on container technologies such as Docker, Cloud Engine strives to provide users with an out-of-the-box experience without having to configure the container environment, write build scripts, and collect logs and statistics. If you want to get the benefits of smooth deployment, fast rollback, and automatic expansion brought by containerization, but don't want to spend time configuring, you might as well try Cloud Engine.

Other references:

• https://www.tutorialworks.com/difference-docker-containerd-runc-crio-oci/

Photo by Sigmund on Unsplash.