Have you ever used these "vaccine" for your cloud native applications?

​Recently, "cloud native" has become a proper "net celebrity" concept in the field of enterprise application development and IT operation and maintenance. With the help of flexible microservice architecture and containerized operation and maintenance methods, as well as continuous automated construction and release capabilities, declarative APIs, and the support of service grid technologies, cloud-native applications have completely changed the design, development, and development of enterprises in many aspects. The way of publishing and operating and maintaining applications has enabled cloud-based companies to obtain significant benefits in terms of flexibility, flexibility, and cost.
In fact, the concept of "cloud native" is still relatively new, and with the evolution of technology and requirements, its related definitions are still iteratively updated. . If you missed the children's shoes, you can go to https://mp.weixin.qq.com/s/-NVUEJK_-gwEMVA-hI4H4A to look back at 16129c2d9a3d35. In this article, we will take a look at the security and protection of containers together, and by sharing security design practices, tell you how to strengthen the security of the container itself and create a solid and reliable foundation for cloud-native applications.

There is an interactive gift at the end of the article!

Containers | Status of Use and Safety

First, let's take a look at an earlier statistical result:

At that time, Docker still occupied a major share. However, it should not be ignored that the share of Containerd in containerization has increased substantially at that time. At present, the Kubernetes community version has changed its support for Docker to "Deprecated" after version 1.20, and will completely remove Docker support after version 1.23.

But Azure Kubernetes Service is faster. As early as version 1.19, Containerd has been used to replace Docker. For users, we inevitably need to go through the migration process, including replacing Docker with the criCtl tool, and changing the location and form of logs.

In summary: RKT, LXC, Mesos and other container runtimes are rare. Docker and Containerd have basically become the mainstream implementation of container runtime .

In addition, in terms of orchestration tools, there is no suspense that Kubernetes has taken the top spot. As shown in the figure below, it can be seen that with the addition of OpenShift and Rancher, its share is as high as 89%, while Swarm has dropped from 11% in 2018 to 5%.

It can be seen from the above statistics that the development trend of dominant technology stacks such as containers and container orchestration is already obvious, but container security is not so optimistic. There are some very shocking data worthy of alert for each of us:

1. 60% of respondents’ organizations have encountered security incidents in the past year
   
2. 93% of respondents are not very clear about the status of the problem
   
3. 82% of respondents believe that due to the adoption of containers, security responsibilities need to be rethinked
   

So for container technology, what security challenges does it pose for us? What kind of response should we take?

Security challenges brought by containers and countermeasures

The container is still a virtual machine, this is a problem:

A container is a "package" that contains all the elements needed to run an application in any environment. The container can be anywhere (private data center, public cloud, developer's personal computer) by virtualizing the operating system. But its degree of virtualization is quite different from that of virtual machines.

It can be seen from the comparison of the following figure: traditional virtual machine occupies a lot of disk space, in addition to the application hosted by the virtual machine, it also contains a complete operating system and related tools; the container is relatively light and only includes running containerized applications The required libraries and tools are therefore more compact than virtual machines and start faster .

The main technical realization of the container:

Taking Docker container as an example, these three features are mainly used to realize virtualization and isolation: cgroup + namespace + docker image:

cgroup (resource limit):

driver :

• cgroupfs:

To limit memory, CPU and other resources to limit writing to a cgroup file corresponding to pid

• systemd :

Provide a cgroup management method. All cgroup writing operations must be done through the systemd interface, and the cgroup files cannot be changed manually.

Commonly used display items:

• CPU: calculation
• Memory: memory
• Device: device
• Freezer: batch task freeze (for safety)
• Blkio: Disk access
• Pid: number of processes created

namespace (isolated):

• mount: The view of the file system, which is a file system provided by the container image.
• uts: Isolate hostname and domain.
• pid namespace: Ensure that the init process of the container is started with process number 1.
• network namespace: Except the host network mode, all other network modes have a networknamespace file.
• user namespace: Controls a mapping between user UID and GID in the container and on the host.
• IPC namespace: Some things that control the process and communication, such as semaphore.

docker image

• Based on the Joint File System
• Different layers can be reused by other mirrors
• The container read-write layer is the latest layer of the latest image

The main security technologies used in containerization:

While the container brings lightness and convenience, it also brings some safety issues. The main reasons include:

• The operating system kernel of the same host is used among multiple containers;
• Many resources and objects in the Linux kernel cannot be Namespaced;
• System call screening has become complicated (although security can be strengthened by filtering and screening all system calls initiated within the container through technologies such as Seccomp, but an additional layer of system call filtering will slow down the performance of the container. In addition, by default, we It is also difficult to know which system calls should be enabled and which system calls should be prohibited).

As everyone pays more and more attention to security, the container security technology itself is constantly strengthening and evolving. Representative technologies include:

• By default, Capability is used to limit the capabilities of container Root users. The Capability of Docker's default container includes (Docker's default Secomp can be found at https://github.com/moby/moby/blob/master/profiles/seccomp/default.json):

• Seccomp system call filtering:
  

• Combine seLinux or Apparmor to implement MAC access control:

Create a safe container:

The essence of container security is the shared kernel. In order to solve this problem, the concept of "secure container" has been gradually developed. Its main representative technologies are Kata Containers and Gvisor.

The essence of Kata Containers is a lightweight virtual machine. When we start a Kata Containers, we will see a normal virtual machine running. This also means: the standard virtual machine manager (Virtual Machine Manager, VMM) is an essential component for running Kata Containers.

In 2018, Google released the gVisor project. The gVisor project configures a very small "independent kernel" implemented in Go language for the container process and runs in user mode. This kernel exposes the Linux kernel to the container process and plays the role of "Guest Kernel", thus achieving the purpose of isolating the container from the host. The general structure comparison between the two is as follows:

Container docker-bench-security:

In a specific production environment, we can execute scripts to obtain best practice verification results. Operation and maintenance personnel can make modifications and remarks based on these suggestions. The detailed address is as follows: https://github.com/docker/docker-bench-security

The following figure is the execution result, from which you can see the entries of each level:

Mirror security scan:

The execution program and file system of the container are mirror images. To ensure the security of the container, the security scan of the mirror image is also very important. Open source image scanning products can choose Clair.

On Azure, the Azure Defender registered by the container includes a vulnerability scanner that scans the mirrors in the Azure Container Registry registry based on Azure Resource Manager. This technology is supported by Qualys, the industry's leading vulnerability scanning provider.

Security issues of Kubernetes

The security issues of Kubernetes can be developed from several different levels.

First of all, Kubernetes itself is a resource object management platform. By abstracting different objects, it forms resource objects that the platform can recognize, such as Pod, Namespace, and ReplicaSet. Access to these objects requires a unified API. There are three main security mechanisms at the API level: authentication, authorization, and access control.

Kubernetes API access control:

Among them, admission control is divided into two kinds of hooks:

1. Mutating admission webhook: make corresponding changes to the resource definition statement;
2. Validating admission webhook: Determine whether the resource can be created or updated.

In terms of API authentication, the most commonly used methods are X509 and ServiceAccount.

X509 is achieved by specifying the corresponding Name and Group attributes:

The Pod’s ServiceAccount authentication method is shown in the figure below:

In terms of authorization, role-based access control mechanisms are mainly adopted. Roles in Kubernetes can be divided into Role and ClusterRole according to whether they need to be accessed across namespaces, and the binding relationship can also be divided into RoleBinding and ClusterRoleBinding. The relationship between them is shown in the following figure:

In addition, pay attention to the security context of Kubernetes. Since the bottom layer of Kubernetes still performs life cycle management and security restrictions on containers by invoking the container runtime, the above-mentioned security thinking about Docker is also applicable to Kubernetes, so that runtime control can be achieved. For details, please refer to the figure below:

In addition to controlling the runtime through the aforementioned securityContext, Kubernetes also introduces the resource security rule object PodSecurityPolicy to achieve unified security control of Pods in the namespace. The main control contents are as follows:

Its workflow is shown in the figure below:

In terms of traffic access control, inbound and outbound traffic in Kubernetes can be defined through NetworkPolicy. Specifically, we can define the following:

ServiceMesh：

If you need multi-tenant or enterprise-level traffic and security control, just using NetworkPolicy is not enough. At this time, ServiceMesh needs to be introduced. Each ServiceMesh can run well on the base of Kubernetes.

Currently popular ServiceMesh mainly includes Istio and Linkerd. The following figure shows the overall architecture and component composition of ServiceMesh represented by Istio. Because it also has certain configuration and management complexity, the performance loss cannot be ignored, so for companies with incomplete technical teams, many companies are still on the sidelines.

OpenAgentPolicy

Since PodSecurityPolicy itself can only define Pod rules, in cloud native applications, including in various public or private clouds, more resource types, such as Service, Node, etc., will be introduced.

The open source community has gradually begun to abandon PodSecurityPolicy, and instead recommends using the OpenAgentPolicy framework to implement constraints on various resources. Because of the need to constrain various resources, it is necessary to provide greater flexibility, and then introduce a new, more expressive rule language: Rego. For an example, please refer to the figure below:

Specific to the Kubernetes platform, we need to use the GateKeeper component.

In the Azure cloud platform, this function is implemented and replaced by Azure Policy:

Storage security

Kubernetes can store secret information such as keys in the Secret resource, but looking at the content in etcd will find that everything is implemented in plaintext, which will lead to some security problems, such as:

Provide services on Azure that store sensitive data and critical business data such as encryption keys, certificates (and private keys associated with certificates) and secrets (such as connection strings and passwords)--Azure Key Vault. It can be seamlessly integrated with Azure Kubernetes Service to maximize the access and storage security of these sensitive information.

Azure Kubernetes Service security

AKS mainly contains the Kubernetes functional modules shown in the figure below, which can be roughly divided into two parts: the control plane and the data plane:

Azure Kubernetes Service is a managed Kubernetes service based on the Azure cloud platform. It turns the control plane into a hosted platform service. Customers do not need to manage API Servers, Etcd and other components, but only need to pay for their own Agent node. At the same time, in terms of security, the Azure platform provides services such as Application Gateway, Azure Firewall, and Azure Key Vault to ensure system security.

The following figure is a relatively common, overall architecture diagram based on Azure Kubernetes Service:

The whole structure is divided into four parts:

1. Container and image security:

• Through AAD certification to ensure the safety of container certification
• Ensure image security through ACR image scanning

2. Node and cluster security:

• Node automatic patch update
• Nodes can be published to the private network

3.Pod security:

• Provide rich security definitions through Azure Policy

4. Flow security:

• Encrypted by storage
• Ensure the security of secret keys etc. through Key Vault
• Ensure access security through Azure Application Gateway, etc.

For more information about AKS, please refer to related documents. At the same time, there are some hands-on experiments in this area. For details, please refer to https://github.com/sme-csu/aks-security-design-practice.

---

The security of containers is a huge topic. Due to space limitations, it is impossible to introduce them in detail in this article. I hope that the above content can play a role in attracting ideas and help everyone to better understand related technologies and design a safer container environment.

If you want to learn more about the related topics covered in this article, you can go to https://github.com/sme-csu/App-Mod to view related presentation documents and video tutorials.

If you have any comments or suggestions, please leave a message to communicate.

Pop quiz

Compared with ordinary containers, what permissions are added to privileged containers? What if we prevent the container from being elevated to a privileged container or use a privileged container?

Share your views in the comment area. We will select a lucky user and send out Azure customized gifts. Come and leave your opinion!

---

Scan the QR code to follow Microsoft MSDN to get more first-hand technical information and official learning materials from Microsoft!