cOS-toolkit: the next step in Container OS

About the Author
Zhang Zhibo, R&D Director of SUSE Rancher Greater China, has been active in the front line of R&D and has experienced the technological transformation from OpenStack to Kubernetes. He has rich R&D and practical experience in the field of underlying operating system Linux, virtualized KVM and Docker container technology.

Background: I once maintained RancherOS, an open source containerized operating system for Docker. As an emerging project that continues the idea of Container OS, cOS-toolkit has made a deeper abstraction. This article will gradually analyze the design concept and usage of cOS-toolkit, and share some personal thoughts.

Reflections on

At the beginning of the release of RancherOS, Docker was in full swing. It is logical that the goal of RancherOS is to become a Container OS for Docker. In addition to the basic standard features of Immutable OS, we have made a lot of magic changes: for example, we changed dockerd to system-docker to replace systemd, so that we can manage some system services like managing containers; we have cut the kernel to the extreme, hoping to make it a Run Docker's OS at the lowest cost; two layers of rootfs, of course, the rootfs of user space is essentially a container running on images such as Ubuntu/CentOS/Buildroot; use YAML configuration list for system initialization, and simplify system management through this abstraction staff burden, etc.

Over time, some ideas of magic change become unsustainable with upstream development. For example, the system-docker component always stays in an early version and cannot evolve. Thinning the kernel also brings many problems. Without a strong community, it is difficult to maintain these kernels and system components continuously and provide a credible guarantee of compatibility. The default console is built based on the Buildroot project. It is very tedious and complicated to add packages every time, and the simplification pursued by Buildroot itself sometimes cannot meet the needs of the server side. It is difficult to meet the user's customization needs. Users usually need to wait for the official version to be updated. The cost of user-defined technology is very high and it is difficult to get started.

During the development of RancherOS, a certain number of community users have been obtained, and a small range of commercial support has been tried. It turns ideas into reality and is a good open source project for specific groups. However, from the perspective of sustainable development, stopping maintenance is a more rational choice. Although, it can be seen that community users are still releasing some RancherOS derivatives BurmillaOS ( https://burmillaos.org/ ).

cOS-toolkit takes over the next journey

cOS-toolkit is a toolkit for building ContainerOS. Previously, RancherOS itself was a Linux distribution, and cOS-toolkit provides a more powerful abstraction for building such distributions. Drawing on the experience of some previous open source projects, combined with the mainstream needs of the current market, cOS-toolkit can bring the following key features:

• Build and upgrade using the Container method. cOS-toolkit uses the open source tool luet ( https://github.com/mudler/luet ), luet is a container-based multi-platform package management tool. The core capability of cOS-toolkit comes from luet, and cOS-toolkit itself maintains several luet packages ( https://github.com/rancher-sandbox/cOS-toolkit/tree/master/packages ). On top of the base images of various Linux distributions, these packages can be superimposed to build Container OS based on the Dockerfile mode. This approach can rely on the strong community behind each Linux distribution, cOS-toolkit does not need to pay attention to how a certain package is compiled and integrated, and will not break the usage habits of each distribution enthusiast.

• OTA updates are supported. This feature is still evolving. The OTA of Container OS can build and publish the updated content to the mirror warehouse, and use the extended orchestration capabilities of Kubernetes to perform system upgrades for all nodes. The upgrade action triggers the update of specific container images and applies them to In the OS built by cOS-toolkit.

• Cloud-init support, based on systemd, and immutable. The support ability of Cloud-init directly reflects the friendliness of the public cloud environment; based on systemd is an important choice for sustainable development; the support for Immutable has priority in the entire design, which is almost all Container OS. Standard properties.
• Easy and convenient customization. cOS-toolkit is no longer just a distribution, it provides more powerful custom build capabilities. In addition to referencing packages maintained in cOS-toolkit, users can also customize packages for expansion. At the same time, cOS-toolkit itself also simplifies Various OEM configurations ( https://rancher-sandbox.github.io/cos-toolkit-docs/docs/customizing/oem_configuration/ ), changing branding and changing initial username and password is very simple. Powerful customization capabilities also allow users to build specific OSes for hardware specifications such as cloud environments or edge environments. Drivers and software packages for various hardware environments rely on the strong distribution community behind them, regardless of quality or compatibility. Sex is guaranteed.

The cOS-toolkit project builds some base distributions by default:

• Blue: Based on Fedora
• Green: based on openSUSE
• Orange: Based on Ubuntu

Since the core engineers are from SUSE, the openSUSE-based variants will be tested more completely. These mirror media can be downloaded in Github Release ( https://github.com/rancher-sandbox/cOS-toolkit/releases ).

SUSE Rancher has started building some emerging products with cOS-toolkit, such as Harvester OS and RancherOS v2. The former provides the base OS of the Harvester cluster, which is convenient for Harvester to manage nodes at a deeper level; the latter will be committed to providing a base OS for Rancher and K3s/RKE2 in the future, further simplifying product deployment and operation and maintenance difficulties. RancherOS v2 is also actively incubating open source: https://github.com/rancher-sandbox/os2

cOS-toolkit first experience

The friendly experience that supports public cloud has become the standard of emerging operating systems. The Green (openSUSE-based) variant built by cOS-toolkit fully supports public clouds such as AWS/Azure/GCP. We take the AWS version as an example for the initial experience. The AMI information can be downloaded from the Github Release ( https://github.com/rancher-sandbox/cOS-toolkit/releases ). The experience version of this article is v0.7.4.

Since these AMIs are built using UEFI by default, some instance types cannot be supported. This experience uses t3.large.

By default, the system will initially enter Recovery mode. Users can perform custom installation in this mode, and perform system initial installation through the built-in installation tool with the predefined YAML configuration. However, since we are using the AWS cloud environment, we can use the cloud-init mechanism to simplify this process. When launching an instance based on the cOS AMI (root disk 16GiB), configure the following cloud-init stuff:

name: "Default deployment"
stages:
   rootfs.after:
     - name: "Repart image"
       layout:
         # It will partition a device including the given filesystem label or part label (filesystem label matches first)
         device:
           label: COS_RECOVERY
         add_partitions:
           - fsLabel: COS_STATE
             # 10Gb for COS_STATE, so the disk should have at least 16Gb
             size: 10240
             pLabel: state
           - fsLabel: COS_PERSISTENT
             # unset size or 0 size means all available space
             pLabel: persistent
   initramfs:
     - if: '[ -f "/run/cos/recovery_mode" ]'
       name: "Set sshd to wait for deployment"
       files:
       - path: "/etc/systemd/system/sshd.service.d/override.conf"
         content: |
             [Unit]
             After=cos-setup-network.service
   network:
     - if: '[ -f "/run/cos/recovery_mode" ]'
       name: "Deploy cos-system"
       commands:
         - |
             cos-deploy --no-verify --no-cosign && shutdown -r now

This cloud-init userdata will help us plan the root disk partition table information and perform initial installation at the same time. Note that if no version is specified when cos-deploy, the latest version will be installed. This means that although we use the v0.7.4 AMI, after cos-deploy it will look in the source for a newer version to deploy to the root disk, which is actually part of the OTA update mechanism.

After the system is installed and initialized, it can be accessed through ssh. Since the metadata of AWS has not been fully adapted, the user/password method is used to access temporarily, and we have not changed the password in userdata, so the default user name and password root/cos can be used . The whole system will be very clean and concise. Since the Green variant uses the openSUSE distribution as the base, its kernel version is also consistent with openSUSE 15.3, and the system version is also tracked to v0.8.2.

In addition to using the cloud-init mechanism to install the system, you can also manually use cos-deploy to install it. You can clearly see that the newer system image of v0.8.2 has been pulled during the execution process.

Of course, we can also make a custom image and push it to the image repository. The construction of the system image can rely on the Dockerfile mechanism. Take the standard example in Github Repo ( https://github.com/rancher-sandbox/cOS-toolkit/blob/master/examples/standard/Dockerfile ) as an example, after using openSUSE as the base image, execute in the Dockerfile zypper to install packages that rely on a strong community to ensure quality:

ARG LUET_VERSION=0.20.6
FROM quay.io/luet/base:$LUET_VERSION AS luet


FROM opensuse/leap:15.3


ENV COSIGN_EXPERIMENTAL=1
ENV COSIGN_REPOSITORY=raccos/releases-green


ARG ARCH=amd64
ENV ARCH=${ARCH}
RUN zypper in -y \
    bash-completion \
    conntrack-tools \
    coreutils \
    curl \
    device-mapper \
    dosfstools \
    dracut \
    kernel-default \
    kernel-firmware-bnx2 \
    kernel-firmware-i915 \
    kernel-firmware-intel \
    kernel-firmware-iwlwifi \
    kernel-firmware-mellanox \
    kernel-firmware-network \
    kernel-firmware-platform \
    kernel-firmware-realtek \
    less \
…
…

We build this Dockerfile into a container image (eg: niusmallnan/containeros:dev) and push it to Dockerhub. Execute on the AWS VM that has just been started: cos-upgrade --no-verify --reboot -d niusmallnan/containeros:dev , after automatic restart, we can get a new version of the OS, which is the OTA update based on container mode Basic demo. Accessing the virtual machine can see that the OS version information, built-in software packages, and kernel versions have all changed:

cOS-toolkit is still being improved, and there is still room for optimization in many details. cOS-toolkit does not provide a certain distribution continuously, but maintains a tool set for building personalized Container OS. Users can rely on it for OS production and release, and at the same time enjoy the container image-style OTA update mechanism it brings.

For more information, please refer to the official documentation: https://rancher-sandbox.github.io/cos-toolkit-docs/docs/