Analysis of OpenYurt's Yurthub Data Filtering Framework

Author: Ying Jianjian, Xinhua Zhi Cloud Computing Center

OpenYurt is the industry's first non-intrusive edge computing cloud-native open source project. It provides users with an integrated cloud-edge experience through edge autonomy, cloud-edge collaboration, edge unitization, and edge traffic closed-loop capabilities. In Openyurt, the edge network can use the data filtering framework to achieve edge traffic closed-loop capability in different node pools.

Analysis of Yurthub data filtering framework

Yurthub is essentially a layer of kube-apiserver proxy, and a layer of cache is added on top of the proxy to ensure that the local cache can be used to ensure business stability when the edge node is offline, which effectively solves the problem of edge autonomy. Second, it can reduce the load on the API on the cloud caused by a large number of list & watch operations.

Yurthub's data filtering is sent to the kube-apiserver through the pod on the node and the kubelet request through the Load Balancer. The proxy receives the response message and performs data filtering processing, and then returns the filtered data to the requester. If the node is an edge node, it will cache the resources in the response request body locally according to the request type. If it is a cloud node, it will not cache locally considering the network status is good.

Schematic diagram of Yurthub's filtering framework implementation:

Yurthub currently contains four filtering rules. The user-agent, resource, and verb requested by addons are used to determine which filter to filter the corresponding data.

Four filtering rule functions and implementation

ServiceTopologyFilter

Data filtering is mainly for EndpointSlice resources, but the Endpoint Slice feature needs to be supported in Kubernetes v1.18 or above. If it is below version 1.18, it is recommended to use the endpointsFilter filter. When passing through this filter, first find the services resource corresponding to the endpointSlice resource through kubernetes.io/service-name, and then judge whether the service resource has an annotation of openyurt.io/topologyKeys. If it exists, then judge the data filtering rule by the value of this annotation. Finally update the response data and return it to addons.

The values ​​of Annotations fall into two broad categories:

1. kubernetes.io/hostname: only filter out the endpoint ip of the same node

2. openyurt.io/nodepool or kubernetes.io/zone: Obtain the corresponding node pool through this Annotations, and finally traverse the endpointSlice resources, and find the corresponding Endpoints in the endpointSlice object through the kubernetes.io/hostname field in the topology field in the endpointSlice. Then reorganize the Endpoints in the endpointSlice and return it to addons.

Code:

 func (fh *serviceTopologyFilterHandler) reassembleEndpointSlice(endpointSlice *discovery.EndpointSlice) *discovery.EndpointSlice {
   var serviceTopologyType string
   // get the service Topology type
   if svcName, ok := endpointSlice.Labels[discovery.LabelServiceName]; ok {
      svc, err := fh.serviceLister.Services(endpointSlice.Namespace).Get(svcName)
      if err != nil {
         klog.Infof("skip reassemble endpointSlice, failed to get service %s/%s, err: %v", endpointSlice.Namespace, svcName, err)
         return endpointSlice
      }
 
      if serviceTopologyType, ok = svc.Annotations[AnnotationServiceTopologyKey]; !ok {
         klog.Infof("skip reassemble endpointSlice, service %s/%s has no annotation %s", endpointSlice.Namespace, svcName, AnnotationServiceTopologyKey)
         return endpointSlice
      }
   }
 
   var newEps []discovery.Endpoint
   // if type of service Topology is 'kubernetes.io/hostname'
   // filter the endpoint just on the local host
   if serviceTopologyType == AnnotationServiceTopologyValueNode {
      for i := range endpointSlice.Endpoints {
         if endpointSlice.Endpoints[i].Topology[v1.LabelHostname] == fh.nodeName {
            newEps = append(newEps, endpointSlice.Endpoints[i])
         }
      }
      endpointSlice.Endpoints = newEps
   } else if serviceTopologyType == AnnotationServiceTopologyValueNodePool || serviceTopologyType == AnnotationServiceTopologyValueZone {
      // if type of service Topology is openyurt.io/nodepool
      // filter the endpoint just on the node which is in the same nodepool with current node
      currentNode, err := fh.nodeGetter(fh.nodeName)
      if err != nil {
         klog.Infof("skip reassemble endpointSlice, failed to get current node %s, err: %v", fh.nodeName, err)
         return endpointSlice
      }
      if nodePoolName, ok := currentNode.Labels[nodepoolv1alpha1.LabelCurrentNodePool]; ok {
         nodePool, err := fh.nodePoolLister.Get(nodePoolName)
         if err != nil {
            klog.Infof("skip reassemble endpointSlice, failed to get nodepool %s, err: %v", nodePoolName, err)
            return endpointSlice
         }
         for i := range endpointSlice.Endpoints {
            if inSameNodePool(endpointSlice.Endpoints[i].Topology[v1.LabelHostname], nodePool.Status.Nodes) {
               newEps = append(newEps, endpointSlice.Endpoints[i])
            }
         }
         endpointSlice.Endpoints = newEps
      }
   }
   return endpointSlice
}

EndpointsFilter

Perform corresponding data filtering for endpoints resources. First, determine whether the endpoint has a corresponding service, obtain the node pool through the node's label: apps.openyurt.io/nodepool, then obtain all nodes under the node pool, and traverse the resources under endpoints.Subsets to find The Ready pod address and NotReady pod address of the same node pool are reorganized into new endpoints and returned to addons.

 func (fh *endpointsFilterHandler) reassembleEndpoint(endpoints *v1.Endpoints) *v1.Endpoints {
   svcName := endpoints.Name
   _, err := fh.serviceLister.Services(endpoints.Namespace).Get(svcName)
   if err != nil {
      klog.Infof("skip reassemble endpoints, failed to get service %s/%s, err: %v", endpoints.Namespace, svcName, err)
      return endpoints
   }
   // filter the endpoints on the node which is in the same nodepool with current node
   currentNode, err := fh.nodeGetter(fh.nodeName)
   if err != nil {
      klog.Infof("skip reassemble endpoints, failed to get current node %s, err: %v", fh.nodeName, err)
      return endpoints
   }
   if nodePoolName, ok := currentNode.Labels[nodepoolv1alpha1.LabelCurrentNodePool]; ok {
      nodePool, err := fh.nodePoolLister.Get(nodePoolName)
      if err != nil {
         klog.Infof("skip reassemble endpoints, failed to get nodepool %s, err: %v", nodePoolName, err)
         return endpoints
      }
      var newEpSubsets []v1.EndpointSubset
      for i := range endpoints.Subsets {
         endpoints.Subsets[i].Addresses = filterValidEndpointsAddr(endpoints.Subsets[i].Addresses, nodePool)
         endpoints.Subsets[i].NotReadyAddresses = filterValidEndpointsAddr(endpoints.Subsets[i].NotReadyAddresses, nodePool)
         if endpoints.Subsets[i].Addresses != nil || endpoints.Subsets[i].NotReadyAddresses != nil {
            newEpSubsets = append(newEpSubsets, endpoints.Subsets[i])
         }
      }
      endpoints.Subsets = newEpSubsets
      if len(endpoints.Subsets) == 0 {
         // this endpoints has no nodepool valid addresses for ingress controller, return nil to ignore it
         return nil
      }
   }
   return endpoints
}

MasterServiceFilter

The ip and port are replaced for the domain name under services. The scenario of this filter is mainly that the pods on the edge can seamlessly use InClusterConfig to access cluster resources.

 func (fh *masterServiceFilterHandler) ObjectResponseFilter(b []byte) ([]byte, error) {
   list, err := fh.serializer.Decode(b)
   if err != nil || list == nil {
      klog.Errorf("skip filter, failed to decode response in ObjectResponseFilter of masterServiceFilterHandler, %v", err)
      return b, nil
   }
 
   // return data un-mutated if not ServiceList
   serviceList, ok := list.(*v1.ServiceList)
   if !ok {
      return b, nil
   }
 
   // mutate master service
   for i := range serviceList.Items {
      if serviceList.Items[i].Namespace == MasterServiceNamespace && serviceList.Items[i].Name == MasterServiceName {
         serviceList.Items[i].Spec.ClusterIP = fh.host
         for j := range serviceList.Items[i].Spec.Ports {
            if serviceList.Items[i].Spec.Ports[j].Name == MasterServicePortName {
               serviceList.Items[i].Spec.Ports[j].Port = fh.port
               break
            }
         }
         klog.V(2).Infof("mutate master service into ClusterIP:Port=%s:%d for request %s", fh.host, fh.port, util.ReqString(fh.req))
         break
      }
   }
 
   // return the mutated serviceList
   return fh.serializer.Encode(serviceList)
}

DiscardCloudService

This filter targets one of the two types of services, LoadBalancer. Since the edge cannot access resources of the LoadBalancer type, the filter will directly filter out this type of resources. The other is for x-tunnel-server-internal-svc under the kube-system namespace. This service mainly exists in the cloud node to access yurt-tunnel-server, and the edge node will directly filter this service.

 func (fh *discardCloudServiceFilterHandler) ObjectResponseFilter(b []byte) ([]byte, error) {
   list, err := fh.serializer.Decode(b)
   if err != nil || list == nil {
      klog.Errorf("skip filter, failed to decode response in ObjectResponseFilter of discardCloudServiceFilterHandler %v", err)
      return b, nil
   }
 
   serviceList, ok := list.(*v1.ServiceList)
   if ok {
      var svcNew []v1.Service
      for i := range serviceList.Items {
         nsName := fmt.Sprintf("%s/%s", serviceList.Items[i].Namespace, serviceList.Items[i].Name)
         // remove lb service
         if serviceList.Items[i].Spec.Type == v1.ServiceTypeLoadBalancer {
            if serviceList.Items[i].Annotations[filter.SkipDiscardServiceAnnotation] != "true" {
               klog.V(2).Infof("load balancer service(%s) is discarded in ObjectResponseFilter of discardCloudServiceFilterHandler", nsName)
               continue
            }
         }
 
         // remove cloud clusterIP service
         if _, ok := cloudClusterIPService[nsName]; ok {
            klog.V(2).Infof("clusterIP service(%s) is discarded in ObjectResponseFilter of discardCloudServiceFilterHandler", nsName)
            continue
         }
 
         svcNew = append(svcNew, serviceList.Items[i])
      }
      serviceList.Items = svcNew
      return fh.serializer.Encode(serviceList)
   }
 
   return b, nil
}

The current state of the filtering framework

The current filtering framework is relatively rigid. The resource filtering is hard-coded into the code, and only registered resources can be filtered accordingly. In order to solve this problem, the filtering framework needs to be modified accordingly.

solution

Option One:

Customize the filter configuration in the form of parameters or environment variables, but this method has the following disadvantages:

1. If the configuration is complex, you need to write the configuration that needs to be customized into the startup parameters or read the environment variables, such as the following format:

 --filter_serviceTopology=coredns/endpointslices#list,kube-proxy/services#list;watch --filter_endpointsFilter=nginx-ingress-controller/endpoints#list;watch

2. Hot update is not possible. Every time you modify the configuration, you need to restart Yurthub to take effect.

Option II:

1. Use the form of configmap to customize the filtering configuration to reduce configuration complexity. Configuration format (user-agent/resource#list, watch) Multiple resources are separated by commas. As follows:

 filter_endpoints: coredns/endpoints#list;watch,test/endpoints#list;watch
filter_servicetopology: coredns/endpointslices#list;watch
filter_discardcloudservice: ""
filter_masterservice: ""

2. Use the Informer mechanism to ensure that the configuration takes effect in real time

Combining the above two points, we chose solution two in OpenYurt.

Problems encountered during development

The api address of Informer watch at the edge is the proxy address of Yurthub, so Yurthub cannot guarantee that the data in the configmap is normal before starting the proxy port. If the addons request is updated before the configmap data after the startup is completed, the data will be returned to the addons without filtering, which will lead to many unexpected problems.

In order to solve this problem, we need to add WaitForCacheSync to the apporve to ensure that the corresponding filter data can be returned after the data synchronization is completed, but adding WaitForCacheSync to the apporve also directly causes the configmap to be blocked when it is watching, so it is necessary to add a white before WaitForCacheSync. List mechanism, when Yurthub uses list & watch to access configmap, we do not directly filter data. The corresponding code logic is as follows:

 func (a *approver) Approve(comp, resource, verb string) bool {
   if a.isWhitelistReq(comp, resource, verb) {
      return false
   }
   if ok := cache.WaitForCacheSync(a.stopCh, a.configMapSynced); !ok {
      panic("wait for configMap cache sync timeout")
   }
   a.Lock()
   defer a.Unlock()
   for _, requests := range a.nameToRequests {
      for _, request := range requests {
         if request.Equal(comp, resource, verb) {
            return true
         }
      }
   }
   return false
}

Summarize

1. From the above expansion capabilities, it can be seen that YurtHub is not only a reverse proxy with data caching capabilities on edge nodes. Instead, a new layer of encapsulation is added to the application lifecycle management of Kubernetes nodes to provide the core management and control capabilities required for edge computing.

2. YurtHub is not only suitable for edge computing scenarios, but can actually be used as a standing component on the node side, suitable for any scenario using Kubernetes. It is believed that this will also drive YurtHub to develop towards higher performance and higher stability.

Click ​​here​​​ to learn about the OpenYurt project today!​