Use an ingress controller to get through the dubbo+k8s network

background

Since all internal services of the company are run on Alibaba Cloud k8s, the IP reported by the dubbo provider to the registry by default is Pod IP , which means that the dubbo service cannot be called in the network environment outside the k8s cluster. If you develop locally If you need to access the dubbo provider service in k8s, you need to manually expose the service to the external network. Our approach is to expose a SLB IP + custom port for each provider service, and use the DUBBO_IP_TO_REGISTRY and DUBBO_PORT_TO_REGISTRY environment variables provided by dubbo. Register the corresponding SLB IP+ custom port in the registry, so that the local network and k8s dubbo service can be connected, but this method is very troublesome to manage. Each service has to customize a port, and each Ports between services cannot be conflicted, and it is very difficult to manage when there are more services.

So I was thinking can not be like nginx ingress as to achieve a seven proxy + virtual domain to reuse a port, through the target dubbo provider application.name do the corresponding forwarding, so all the services only need to register with a SLB IP+port is enough, which greatly improves the convenience. After one party finds that it is feasible after investigation, it will start!

The project is open source: https://github.com/monkeyWie/dubbo-ingress-controller

Technology pre-research

Ideas

1. First of all, dubbo RPC calls are based on the dubbo protocol by default, so I need to see if there is any message information that can be used for forwarding in the protocol, which is to look for the Host request header similar to the HTTP protocol, if there is one. According to this information, the reverse proxy and the virtual domain name are forwarded, and a dubbo gateway nginx is realized on this basis.
2. The second step is to implement dubbo ingress controller , through the watcher mechanism of k8s ingress to dynamically update dubbo gateway, and then all provider services are forwarded by this service in the same way, and the addresses reported to the registry are also unified. The address of the service.

Architecture diagram

dubbo agreement

First on an official agreement chart:

It can be seen that the header of the dubbo protocol is a fixed 16 bytes. There is no expandable field similar to the HTTP header, nor does it carry the application.name field of the target provider, so I mentioned to the official issue , the official The answer is to use the consumer customize the Filter to put the target provider’s application.name into attachments Here, I have to complain about the dubbo protocol. The extended field is actually placed in body If you want to forward it, you need to parse all the request messages. You can get the desired message only after you finish, but the problem is not big, because it is mainly used for the development environment, and this step can barely be achieved.

k8s ingress

k8s ingress is born for HTTP, but the fields inside are enough, let’s look at a section of ingress configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: user-rpc-dubbo
  annotations:
    kubernetes.io/ingress.class: "dubbo"
spec:
  rules:
    - host: user-rpc
      http:
        paths:
          - backend:
              serviceName: user-rpc
              servicePort: 20880
            path: /

The configuration is the same as http through host for forwarding rules, but host configured with the target provider's application.name , and the backend service is the target provider's service . There is a special one that uses a kubernetes.io/ingress.class annotation, which can be specified For which ingress controller this ingress effective, our dubbo ingress controller will only parse the ingress configuration with the dubbo

Development

The previous technical pre-research went well, and then it entered the development stage.

Consumer Custom Filter

As mentioned earlier, if the target provider’s application.name is to be carried in the request, consumers need to customize the Filter code is as follows:

@Activate(group = CONSUMER)
public class AddTargetFilter implements Filter {

  @Override
  public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
    String targetApplication = StringUtils.isBlank(invoker.getUrl().getRemoteApplication()) ?
        invoker.getUrl().getGroup() : invoker.getUrl().getRemoteApplication();
    // 目标提供者的application.name放入attachment
    invocation.setAttachment("target-application", targetApplication);
    return invoker.invoke(invocation);
  }
}

Here again Tucao about, will initiate a request to obtain metadata when dubbo consumers first visit to this request by invoker.getUrl().getRemoteApplication() not get value by invoker.getUrl().getGroup() in order to get.

dubbo gateway

Here is to develop a dubbo gateway nginx , and implement seven-layer proxy and virtual domain name forwarding. The programming language is directly selected as go. First of all, go does network development and has a low mental burden. There is also a dubbo-go project that can be used directly. The decoder, and go has native k8s sdk support, it's perfect!

The idea is to open a TCP Server , then parse the message requested by attachment , target-application attribute in 061935706188b4, and then reverse proxy to the real dubbo provider service. The core code is as follows:

routingTable := map[string]string{
  "user-rpc": "user-rpc:20880",
  "pay-rpc":  "pay-rpc:20880",
}

listener, err := net.Listen("tcp", ":20880")
if err != nil {
  return err
}
for {
  clientConn, err := listener.Accept()
  if err != nil {
    logger.Errorf("accept error:%v", err)
    continue
  }
  go func() {
    defer clientConn.Close()
    var proxyConn net.Conn
    defer func() {
      if proxyConn != nil {
        proxyConn.Close()
      }
    }()
    scanner := bufio.NewScanner(clientConn)
    scanner.Split(split)
    // 解析请求报文，拿到一个完整的请求
    for scanner.Scan() {
      data := scanner.Bytes()
      // 通过dubbo-go提供的库把[]byte反序列化成dubbo请求结构体
      buf := bytes.NewBuffer(data)
      pkg := impl.NewDubboPackage(buf)
      pkg.Unmarshal()
      body := pkg.Body.(map[string]interface{})
      attachments := body["attachments"].(map[string]interface{})
      // 从attachments里拿到目标提供者的application.name
      target := attachments["target-application"].(string)
      if proxyConn == nil {
        // 反向代理到真正的后端服务上
        host := routingTable[target]
        proxyConn, _ = net.Dial("tcp", host)
        go func() {
          // 原始转发
          io.Copy(clientConn, proxyConn)
        }()
      }
      // 把原始报文写到真正后端服务上，然后走原始转发即可
      proxyConn.Write(data)
    }
  }()
}

func split(data []byte, atEOF bool) (advance int, token []byte, err error) {
    if atEOF && len(data) == 0 {
        return 0, nil, nil
    }

    buf := bytes.NewBuffer(data)
    pkg := impl.NewDubboPackage(buf)
    err = pkg.ReadHeader()
    if err != nil {
        if errors.Is(err, hessian.ErrHeaderNotEnough) || errors.Is(err, hessian.ErrBodyNotEnough) {
            return 0, nil, nil
        }
        return 0, nil, err
    }
    if !pkg.IsRequest() {
        return 0, nil, errors.New("not request")
    }
    requestLen := impl.HEADER_LENGTH + pkg.Header.BodyLen
    if len(data) < requestLen {
        return 0, nil, nil
    }
    return requestLen, data[0:requestLen], nil
}

dubbo ingress controller implementation

dubbo gateway has been implemented before, but the virtual domain name forwarding configuration ( routingTable in the code. What we need to do now is to dynamically update this configuration k8s ingress

First, briefly explain ingress controller Take our commonly used nginx ingress controller as an example. It also monitors the k8s ingress resource changes, and then dynamically generates the nginx.conf file. When the configuration is found to be changed, it triggers the nginx -s reload to reload the configuration file.

The core technology used inside is informers , use it to monitor k8s resources, sample code:

// 在集群内获取k8s访问配置
cfg, err := rest.InClusterConfig()
if err != nil {
  logger.Fatal(err)
}
// 创建k8s sdk client实例
client, err := kubernetes.NewForConfig(cfg)
if err != nil {
  logger.Fatal(err)
}
// 创建Informer工厂
factory := informers.NewSharedInformerFactory(client, time.Minute)
handler := cache.ResourceEventHandlerFuncs{
  AddFunc: func(obj interface{}) {
    // 新增事件
  },
  UpdateFunc: func(oldObj, newObj interface{}) {
    // 更新事件
  },
  DeleteFunc: func(obj interface{}) {
    // 删除事件
  },
}
// 监听ingress变动
informer := factory.Extensions().V1beta1().Ingresses().Informer()
informer.AddEventHandler(handler)
informer.Run(ctx.Done())

The forwarding configuration is dynamically updated by implementing the above three events. Each event will carry the corresponding Ingress object information, and then perform the corresponding processing:

ingress, ok := obj.(*v1beta12.Ingress)
if ok {
  // 通过注解过滤出dubbo ingress
  ingressClass := ingress.Annotations["kubernetes.io/ingress.class"]
  if ingressClass == "dubbo" && len(ingress.Spec.Rules) > 0 {
    rule := ingress.Spec.Rules[0]
    if len(rule.HTTP.Paths) > 0 {
      backend := rule.HTTP.Paths[0].Backend
      host := rule.Host
      service := fmt.Sprintf("%s:%d", backend.ServiceName+"."+ingress.Namespace, backend.ServicePort.IntVal)
      // 获取到ingress配置中host对应的service，通知给dubbo网关进行更新
      notify(host,service)
    }
  }
}

docker image provided

k8s above all services need to run in the container, there is no exception, we need dubbo ingress controller constructed docker mirror, constructed here by a two-stage optimization to reduce the volume of the mirror:

FROM golang:1.17.3 AS builder
WORKDIR /src
COPY . .
ENV GOPROXY https://goproxy.cn
ENV CGO_ENABLED=0
RUN go build -ldflags "-w -s" -o main cmd/main.go

FROM debian AS runner
ENV TZ=Asia/shanghai
WORKDIR /app
COPY --from=builder /src/main .
RUN chmod +x ./main
ENTRYPOINT ["./main"]

yaml template provided

To access the k8s API in the cluster, you need to authorize the Pod, authorize it through K8S rbac , and Deployment . The final template is as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: dubbo-ingress-controller
  namespace: default

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dubbo-ingress-controller
rules:
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dubbo-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dubbo-ingress-controller
subjects:
  - kind: ServiceAccount
    name: dubbo-ingress-controller
    namespace: default

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: dubbo-ingress-controller
  labels:
    app: dubbo-ingress-controller
spec:
  selector:
    matchLabels:
      app: dubbo-ingress-controller
  template:
    metadata:
      labels:
        app: dubbo-ingress-controller
    spec:
      serviceAccountName: dubbo-ingress-controller
      containers:
        - name: dubbo-ingress-controller
          image: liwei2633/dubbo-ingress-controller:0.0.1
          ports:
            - containerPort: 20880

If needed later, it can be made into Helm for management.

postscript

So far, the dubbo ingress controller complete. It can be said that the sparrow is small but complete, which involves the dubbo protocol, TCP protocol, seven-layer proxy, k8s ingress , docker and many other things. A lot of this knowledge is in the cloud native What needs to be mastered in the popular era, after the development, I feel that I have benefited a lot.

The complete tutorial can be viewed through github .

Reference link:

• dubbo protocol
• dubbo-go
• Use multiple-ingress-controllers
• Use Golang to customize Kubernetes Ingress Controller

I am MonkeyWie , welcome to scan the code 👇👇 follow! From time to time, share the dry goods knowledge such as JAVA , Golang , front end, docker , k8s