In-depth conversation with the container service ACK distribution second bullet: how to build a hybrid cloud unified network plane with hybridnet

Authors: Ruo He, Yu Sheng, Yu Jia

Reporter: Hello, readers and friends of Alibaba Cloud Native, welcome to the series of exclusive interviews to explore the mystery of life experience again. Today, I invite you old friends, "Alibaba Cloud Container Service ACK Release", last time In the interview, it introduced us to Alibaba's open source cluster mirroring technology sealer, and how they cooperate with each other to achieve fast and stable delivery of Alibaba Cloud ACK services. Readers who missed it, don't forget to review it~ So this time as a guest, who will it introduce to us?

Cloud Container Service ACK Release (ACK Distro for short) : Hello everyone, we meet again! I believe that after the last self-introduction, you already have a general understanding of me, so I won't go into details this time. In this interview, I will continue to explain in detail my good partner: Alibaba's open source Kubernetes container network solution hybridnet, and how I use it to build a hybrid cloud unified network plane.

Reporter from : Okay, then please talk about what hybridnet is and what kind of design concept the project team members came up with to create hybridnet.

Definition of hybridnet and its design philosophy

ACK Distro: Okay, first of all, hybridnet is a Kubernetes container network solution open sourced by Alibaba for hybrid cloud scenarios. It can help users build a unified network plane with a layer of underlay + overlay on the heterogeneous environment of physical machines and virtual machines, and provide rich management, control, operation and maintenance capabilities. At the same time, a new solution is proposed for the container network deployment and operation and maintenance problems in the process of cluster deployment and application delivery in the hybrid cloud scenario.

Its basic design guidelines are:

1. In order to determine a unified network model, reduce cognitive and maintenance costs, and ensure stable long-term evolution
2. Shield the underlying heterogeneous infrastructure, improve the robustness of delivery landing, and reduce production delivery and PoC costs
3. Under the constraints of the unified model, it can not only provide an underlay high-performance pass-through network solution to meet the dual requirements of network connectivity and performance, but also support the provision of an overlay virtual network solution in some performance-insensitive scenarios
4. Minimize the dependence on the external environment and ensure the simplicity of the data plane
5. Deep integration with Kubernetes, providing high-level IPAM capabilities such as dual-stack, IP retention, and IP fixation, to ensure that users’ usage habits remain unchanged after going to the cloud

Different from container network solutions such as terway and aws-cni that are bound to the public cloud or private cloud base of a single IaaS vendor, the project team members hope that hybridnet can solve the consistency and adaptability brought by heterogeneous bases in multi-cloud and hybrid cloud scenarios. It provides agile, general and stable delivery capabilities in different basic network environments, and solves network planning, management, and operation and maintenance problems in complex scenarios through model constraints and operation and maintenance management and control from a unified perspective.

Reporter : Can I understand that? Hybridnet strives to achieve "underlay/overlay hybrid deployment" and "underlay/overlay unified management and operation".

ACK Distro: Yes yes, hybridnet does. I can further expand the description. In a Kubernetes cluster using hybridnet, there can be both underlay and overlay Pods on the same node, and the internal access behavior of all Pods in the cluster is exactly the same without any additional awareness. In this way, users can freely choose and convert between "pure overlay cluster", "pure underlay cluster", and "underlay hybrid cluster", while enjoying the high performance and network direct connection capability brought by the underlay network, as well as the unlimited resources of the overlay network and high adaptability. Moreover, under the constraints of the unified model, the underlay network and the overlay network also maintain the same concepts in terms of management and operation and maintenance.

reporter: Apart from its design concept, can you use a more intuitive method to let readers understand what hybridnet can do?

ACK Distro: Of course, let me explain its functional properties by introducing its core model~ In order to enable hybridnet users to initialize different core models to describe the basic network environment flexibly and diversely, so that the container network can be described in different ways. By abstracting the concepts in the classic network, the project team members introduced the following three core CRD models to abstract and manage network resources.

The core model of hybridnet

Network

In the hybridnet, each Network represents a "scheduling domain", a scheduling domain represents a group of nodes with the same network properties, and Network is the main entry for incoming environmental topology information. A specific IP can move freely among the nodes in the scheduling domain to which it belongs.

Network is associated with nodes through nodeSelector. For some special Networks, such as overlay Network, nodeSelector may be empty. The scheduling domain of this kind of Network is all nodes in the cluster.

Subnet

In hybridnet, Subnet represents the IP resources that can be allocated in a scheduling domain, and Subnet is the main entrance for incoming environmental network IP resource planning information. Each Subnet must belong to a Network. Subnet has more flexible properties:

• Supports the selection of cidr's assignable address range, and can accurately divide a small network segment from a cidr through spec.range.start and spec.range.end
• Support to reserve discrete IP addresses without allocation. When IP addresses are already used in the network segment, you can fill in the used IPs into the array field of spec.range.excludeIPs, and hybridnet will not add these IP addresses. IP assigned to Pod
• Support to reserve the specified IP from being used. When the IP of some Pods needs to be reserved in the network segment and are only designated for use by a specific Pod, these IPs can be filled in the array field of spec.range.reservedPs, and the hybridnet will not Then use these IPs for the allocation of non-designated IPs

IPInstance

IPInstance is currently only used for monitoring, and each IPInstance represents an IP that has actually been allocated to the container network. Through kubectl get IPInstance, you can see the corresponding Pod, Subnet to which it belongs, the node corresponding to the Pod, and so on.

Reporter : So how does the advantage of hybridnet reflect on you? In other words, how do you manage your hybridnet for best practices?

How to manage hybridnet in ACK Distro

ACK Distro: me give you an example of network management operation by operating the above CRD model~ Among them, hybridnet will be deployed as my only built-in network plug-in. (Of course, it is also feasible to customize third-party network plug-ins through the ability of sealer. You can refer to the first article in this series.)

Default behavior

As my fixed behavior, there must be an overlay Network during initialization, and the default network type is overlay at this time. You can view the Network and Subnet information at this time with the following commands:

[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get network
NAME        NETID   SWITCHID
network-0   4       virtual-switch
[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get subnet
NAME                 VERSION   CIDR            START   END   GATEWAY      TOTAL   USED   AVAILABLE   NETID   NETWORK
subnet-0-network-0   4         100.64.0.0/16                 100.64.0.1   65533   2      65531               network-0

As you can see, I will create a Network named network-0 and a Subnet named subnet-0-network-0 after initializing with the default configuration. The CIDR of the container network segment is 100.64.0.0/16. At this time, the newly created Pod will be pulled up in an overlay manner.

Because my basic components have no special network requirements, the biggest advantage of this is that the overlay network helps me shield the underlying network infrastructure. It does not affect the possibility of subsequent network expansion.

From the delivery experience of the hybrid cloud environment, this method can delay the network planning (mainly the underlay network) to the "operation and maintenance" stage, which can minimize the landing cost in the "delivery" stage and improve the deployment efficiency.

Add underlay network

If there are some underlay network requirements (such as "overlay performance bottleneck" or "pod IP's ability to directly reveal to the outside world"), the underlay Pod accounts for a small proportion, especially when you do not want to occupy IP resources in the basic network environment. You can choose to add an additional underlay Network and the corresponding Subnet in addition to the overlay Network created by the default behavior. (The old and new overlay/underlay Network does not have any dependency order on the model)

In the experimental environment of this example, the node network segment is 192.168.56.0/24 (all nodes are in a classic Layer 2 network), because the node IP only uses 192.168.56.1, 192.168.56.2, 192.168.56.3, 192.168. 56.4, we consider leaving the unused 192.168.56.100 to 192.168.56.150 address range for the container to use to build the simplest underlay network. In this case we only need to apply the following yaml:

---
apiVersion: networking.alibaba.com/v1
kind: Network
metadata:
  name: underlay-network1
spec:
  netID: 0
  nodeSelector:
    network: network1
  type: Underlay

---
apiVersion: networking.alibaba.com/v1
kind: Subnet
metadata:
  name: underlay-subnet1
spec:
  network: underlay-network1
  netID: 0
  range:
    version: "4"
    cidr: "192.168.56.0/24"
    gateway: "192.168.56.254"
    start: "192.168.56.100"
    end: "192.168.56.150"

Because Network is associated with Node through nodeSelector, we need to label the node where we want to deploy underlay Pod with the corresponding Network nodeSelector label. Here we only want to have underlay type Pod on node izf8zdygpbo4hx57g2wah8z:

kubectl label node izf8zdygpbo4hx57g2wah8z network=network1

At this time, the default network type is still the overlay network. To create an underlay Pod, you only need to simply add the annotation specification of networking.alibaba.com/network-type: Underlay to the Pod. The effect is as shown in the figure:

[root@iZf8zdygpbo4hx57g2wahaZ ~]# kubectl get po -owide -n test
NAME                                 READY   STATUS    RESTARTS   AGE     IP               NODE                      NOMINATED NODE   READINESS GATES
curl-deployment-1-5cfb5dcb8c-65fr7   1/1     Running   0          11m     100.64.0.29      izf8zdygpbo4hx57g2wahbz   <none>           <none>
curl-deployment-1-5cfb5dcb8c-hp626   1/1     Running   0          11m     100.64.0.26      izf8zdygpbo4hx57g2wahbz   <none>           <none>
curl-deployment-1-5cfb5dcb8c-qbr6w   1/1     Running   0          11m     100.64.0.27      izf8zdygpbo4hx57g2wah7z   <none>           <none>
curl-deployment-1-5cfb5dcb8c-zclv2   1/1     Running   0          11m     100.64.0.31      izf8zdygpbo4hx57g2wahbz   <none>           <none>
curl-deployment-1-5cfb5dcb8c-zfqkp   1/1     Running   0          11m     100.64.0.28      izf8zdygpbo4hx57g2wah7z   <none>           <none>
curl-ss-0                            1/1     Running   0          6m24s   192.168.56.140   izf8zdygpbo4hx57g2wah8z   <none>           <none>
curl-ss-1                            1/1     Running   0          6m5s    192.168.56.141   izf8zdygpbo4hx57g2wah8z   <none>           <none>
curl-ss-2                            1/1     Running   0          6m1s    192.168.56.142   izf8zdygpbo4hx57g2wah8z   <none>           <none>

Modify the default network type to underlay

If the network demands of your underlay account for the vast majority, you want to create an overlay Pod by default. You can also modify the default network type to underlay. After the modification, the Pod will be created with the underlay network by default, and you can still add it to the Pod by adding The way of annotation specifies that the Pod is created with the overlay network. Overlay Pods already created will not be affected.

To modify the default network type, you need to kubectl edit deploy hybridnet-webhook -n kube-system and kubectl edit deploy hybridnet-manager -n kube-system respectively, and modify the DEFAULT_NETWORK_TYPE environment variable of container startup to Underlay:

spec:
  containers:
    - name: hybridnet-[manager|webhook]           
      command:
        - /hybridnet/hybridnet-[manager|webhook]
      env:
        - name: DEFAULT_NETWORK_TYPE
          # "Overlay" or "Underlay", 
          # default "Underlay" if environment variable not configured. 
          value: Underlay

After this modification, the Pod will be created in the underlay mode by default, and the network connectivity between the new underlay Pod and the original overlay Pod will not be affected (in simple terms, the equivalent underlay Pod will have a communication with other overlay Pods. the identity of the overlay). * *

Add/remove network resources

As shown in the above example, adding the network resources of Network/Subnet only needs to apply the yaml corresponding to the CR. Once the Network/Subnet is applied, the hybridnet will consider that the basic network configuration has been completed in the environment, and use the corresponding CR for network resources. distribute.

From a security point of view, the operation of deleting a Network/Subnet is subject to basic constraints. The Subnet itself can be deleted only when no IP in the Subnet is in use; similarly, the Network itself can be deleted only by deleting all the Subnets in the Network first.

ACK Distro : All in all, with the help of hybridnet, I can make Alibaba Cloud Container Service ACK build a unified network plane of underlay + overlay on top of heterogeneous environments, improve management, control, operation and maintenance capabilities, and bring more benefits to developers. Good container service experience.

Reporter from : Okay, thank you very much for your careful explanation this time. After the second in-depth interview, I will say goodbye to you again. I look forward to meeting readers and friends next time.

ACK Distro: you next time!

Related Links​

[1] hybridnet open source warehouse address

​https://github.com/alibaba/hybridnet​​

[2] hybridnet community documentation

​https://github.com/alibaba/hybridnet/wiki​​

[3]ACK Distro official website

​https://www.aliyun.com/product/aliware/ackdistro​​

[4] ACK Distro Official GitHub​[​]

​​https://github.com/AliyunContainerService/ackdistro​​

Recommended in the past

​1. Make innovation at your fingertips, Alibaba Cloud Container Service ACK release version is open for free download

​2. The first in-depth dialogue with the ACK release of Alibaba Cloud Container Service: How to use sealer to achieve rapid build & deployment

---

For more information, please scan the QR code below or search WeChat account (AlibabaCloud888) to add cloud native assistant ! Get more information about it!

​​