Understand SuperEdge Cloud Side Tunnel in one article

Author

Li Tengfei, Tencent container technology R&D engineer, Tencent Cloud TKE background R&D, SuperEdge core development member.

Du Yanghao, a senior engineer at Tencent Cloud, is passionate about open source, containers and Kubernetes. Currently, he is mainly engaged in mirror warehouse, Kubernetes cluster high availability & backup and restoration, and edge computing related research and development.

SuperEdge introduction

SuperEdge is a native edge container solution for Kubernetes. It extends the powerful container management capabilities of Kubernetes to edge computing scenarios, and provides solutions to common technical challenges in edge computing scenarios, such as: single cluster nodes across regions, cloud edge networks are not Reliable, edge nodes are located in the NAT network, etc. These capabilities allow applications to be easily deployed on edge computing nodes and run reliably. They can help you easily manage distributed computing resources in a Kubernetes cluster, including but not limited to: edge cloud computing Resources, private cloud resources, on-site equipment, create your edge PaaS platform. SuperEdge supports all Kubernetes resource types, API interfaces, usage methods, and operation and maintenance tools. There is no additional learning cost. It is also compatible with other cloud-native projects, such as Promethues. Users can use it in combination with other cloud-native projects that they need. The project was co-sponsored by the following companies: Tencent, Intel, VMware, Huya Live, Cambrian, Capital Online, and Meituan.

The architecture and principle of the cloud side tunnel

In edge scenarios, it is often a one-way network, that is, only edge nodes can actively access the cloud. The cloud edge tunnel is mainly used to proxy the request of the cloud to access the edge node components, and solve the problem that the cloud cannot directly access the edge node.

The architecture diagram is as follows:

The realization principle is:

• tunnel-edge on the edge node actively connects to the tunnel-cloud service, and the tunnel-cloud service transfers the request to the tunnel-cloud pod according to the load balancing policy
• tunnel-edge and tunnel-cloud establish a gRPC connection, tunnel-cloud will write the mapping of its podIp and the nodeName of the node where tunnel-edge tunnel-dns . gRPC is disconnected, tunnel-cloud will delete the mapping of related podIp and node name

The proxy forwarding process of the entire request is as follows:

• When kubelet or other applications on the application access edge node apiserver or other clouds, Tunnel-DNS through DNS hijacking (the the HTTP node name in the Request host resolves to a Tunnel-Cloud podIp) of the request Forward to the pod of tunnel-cloud
• tunnel-cloud forwards the request information according to the node name to the tunnel-edge gRPC connection established by the node name
• tunnel-edge requests the application on the edge node according to the received request information

Tunnel internal module data exchange

After introducing the configuration of the Tunnel, the following describes the internal data flow of the Tunnel:

The above figure marks the data flow of HTTPS proxy. The data flow of TCP proxy is similar to that of HTTPS. The key steps are:

• HTTPS Server -> StreamServer: HTTPS Server sends StreamMsg to Stream Server through Channel, where Channel is to obtain node.Channel from nodeContext according to StreamMsg.Node field
• StreamServer -> StreamClient: Each cloud side tunnel will allocate a node object, and send the StreamClient to the Channel in the node to send the data to the StreamClient
• StreamServer -> HTTPS Server: StreamServer sends StreamMsg to HTTPS Server through Channel, where Channel is to obtain node from nodeContext according to StreamMsg.Node, and to obtain conn.Channel of HTTPS module by matching StreamMsg.Topic with conn.uid

nodeContext and connContext are both for connection management, but gRPC long connection and connContext management upper layer forwarding request connection ( TCP and HTTPS ) are different, so they need to be managed separately.

Tunnel connection management

Tunnel management connections can be divided into bottom-level connections (gRPC connections of cloud tunnels) and upper-level application connections (HTTPS connections and TCP connections). The management of abnormal connections can be divided into the following scenarios:

The gRPC connection is normal, but the upper layer connection is abnormal

Taking HTTPS connection as an example, tunnel-edge is abnormally disconnected from the edge node Server, and a StreamMsg (StreamMsg.Type=CLOSE) message will be sent. tunnel-cloud will actively close HTTPS after receiving the StreamMsg message. The connection between Server and HTTPS Client.

gRPC connection abnormal

If the gRPC connection is abnormal, the Stream module will send StreamMsg (StreamMsg.Type=CLOSE) to the HTTPS and TCP modules according to the node.connContext bound to the gPRC connection, and the HTTPS or TCP module will actively disconnect after receiving the message.

Stream (gRPC cloud side tunnel)

func (stream *Stream) Start(mode string) {
    context.GetContext().RegisterHandler(util.STREAM_HEART_BEAT, util.STREAM, streammsg.HeartbeatHandler)
    if mode == util.CLOUD {
        ...
        //启动gRPC server
        go connect.StartServer()
        ...
        //同步coredns的hosts插件的配置文件
        go connect.SynCorefile()
    } else {
        //启动gRPC client
        go connect.StartSendClient()
        ...
    }
    ...
}

tunnel-cloud First call RegisterHandler to register the heartbeat message processing function HeartbeatHandler
SynCorefile executes to synchronize tunnel-coredns . CheckHosts is executed once every minute (considering the time for configmap to synchronize the pod mount file of tunnel-cloud

func SynCorefile() {
    for {
        ...
        err := coreDns.checkHosts()
        ...
        time.Sleep(60 * time.Second)
    }
}

And checkHosts is responsible for the specific refresh operation of configmap:

func (dns *CoreDns) checkHosts() error {
    nodes, flag := parseHosts()
    if !flag {
        return nil
    }
    ...
    _, err = dns.ClientSet.CoreV1().ConfigMaps(dns.Namespace).Update(cctx.TODO(), cm, metav1.UpdateOptions{})
    ...
}

checkHosts first calls parseHosts to obtain the name of the edge node in the local hosts file and the corresponding tunnel-cloud podIp mapping list, compare the corresponding node name of podIp with the node name in memory, if there is a change, overwrite this content into the configmap and update:

In addition, the purpose of introducing the configmap local mount file into the tunnel-cloud here is to optimize the performance of the hosting mode when many clusters simultaneously synchronize the tunnel-coredns

tunnel-edge first call StartClient to establish a gRPC tunnel-edge , and return grpc.ClientConn

func StartClient() (*grpc.ClientConn, ctx.Context, ctx.CancelFunc, error) {
    ...
    opts := []grpc.DialOption{grpc.WithKeepaliveParams(kacp),
    grpc.WithStreamInterceptor(ClientStreamInterceptor),
    grpc.WithTransportCredentials(creds)}
    conn, err := grpc.Dial(conf.TunnelConf.TunnlMode.EDGE.StreamEdge.Client.ServerName, opts...)
    ...
}

Passed when calling grpc.Dial grpc.WithStreamInterceptor(ClientStreamInterceptor) DialOption, is transmitted to the ClientStreamInterceptor grpc.ClientConn as StreamClientInterceptor, waiting GRPC connected state to the Ready, then a Send function. streamClient.TunnelStreaming calls StreamClientInterceptor and returns a wrappedClientStream object

func ClientStreamInterceptor(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) {
    ...
    opts = append(opts, grpc.PerRPCCredentials(oauth.NewOauthAccess(&oauth2.Token{AccessToken: clientToken})))
    ...
    return newClientWrappedStream(s), nil
}

ClientStreamInterceptor will construct the edge node name and token into oauth2.Token.AccessToken for authentication transmission, and construct wrappedClientStream

stream.Send will concurrently call wrappedClientStream.SendMsg and wrappedClientStream.RecvMsg to tunnel-edge , and block waiting

Note: Tunnel-edge registers node information with tunnel-cloud when creating gRPC Stream, not when creating grpc.connClient

The whole process is shown in the figure below:

Correspondingly, when initializing tunnel-cloud , grpc.StreamInterceptor(ServerStreamInterceptor) will be constructed as gRPC ServerOption, and ServerStreamInterceptor will be passed to grpc.Server as StreamServerInterceptor:

func StartServer() {
    ...
    opts := []grpc.ServerOption{grpc.KeepaliveEnforcementPolicy(kaep), grpc.KeepaliveParams(kasp), grpc.StreamInterceptor(ServerStreamInterceptor), grpc.Creds(creds)}
    s := grpc.NewServer(opts...)
    proto.RegisterStreamServer(s, &stream.Server{})
    ...
}

When the cloud gRPC service receives a tunnel-edge request (to establish a Stream stream), it will call the ServerStreamInterceptor, and the ServerStreamInterceptor will 160aca781346e4 gRPC metadata from the 160aca781346e4 gRPC metadata 160aca781346e5 and parse the gRPC gRPC 160aca781346e7 corresponding edge The token is verified, and then a wrappedServerStream is constructed according to the node name as the processing object for communicating with the edge node (each edge node corresponds to a processing object), the handler function will call stream.TunnelStreaming and pass the wrappedServerStream to it (wrappedServerStream implements proto.Stream_TunnelStreamingServer interface)

func ServerStreamInterceptor(srv interface{}, ss grpc.ServerStream, info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
    md, ok := metadata.FromIncomingContext(ss.Context())
    ...
    tk := strings.TrimPrefix(md["authorization"][0], "Bearer ")
    auth, err := token.ParseToken(tk)
    ...
    if auth.Token != token.GetTokenFromCache(auth.NodeName) {
        klog.Errorf("invalid token node = %s", auth.NodeName)
        return ErrInvalidToken
    }
    err = handler(srv, newServerWrappedStream(ss, auth.NodeName))
    if err != nil {
        ctx.GetContext().RemoveNode(auth.NodeName)
        klog.Errorf("node disconnected node = %s err = %v", auth.NodeName, err)
    }
    return err
}

When the TunnelStreaming method exits, it will execute ServerStreamInterceptor to remove the node logic ctx.GetContext().RemoveNode

TunnelStreaming will concurrently call wrappedServerStream.SendMsg and wrappedServerStream.RecvMsg to tunnel-cloud , and block waiting:

func (s *Server) TunnelStreaming(stream proto.Stream_TunnelStreamingServer) error {
    errChan := make(chan error, 2)
    go func(sendStream proto.Stream_TunnelStreamingServer, sendChan chan error) {
        sendErr := sendStream.SendMsg(nil)
        ...
        sendChan <- sendErr
    }(stream, errChan)
    go func(recvStream proto.Stream_TunnelStreamingServer, recvChan chan error) {
        recvErr := stream.RecvMsg(nil)
        ...
        recvChan <- recvErr
    }(stream, errChan)
    e := <-errChan
    return e
}

SendMsg will receive StreamMsg from the edge node node corresponding to wrappedServerStream, and call ServerStream.SendMsg to send the message to tunnel-edge

func (w *wrappedServerStream) SendMsg(m interface{}) error {    if m != nil {        return w.ServerStream.SendMsg(m)    }    node := ctx.GetContext().AddNode(w.node)    ...    for {        msg := <-node.NodeRecv()        ...        err := w.ServerStream.SendMsg(msg)        ...    }}

RecvMsg while constantly receiving from Tunnel-Edge StreamMsg and calls StreamMsg. Handler corresponding to operate

summary:

• The Stream module is responsible for establishing gRPC connection and communication (cloud side tunnel)
• tunnel-edge on the edge node actively connects to the cloud tunnel-cloud service, tunnel-cloud service, according to the load balancing policy, the request is forwarded to the tunnel-cloud pod
• tunnel-edge and tunnel-cloud establish a gRPC connection, the tunnel-cloud will write its own podIp and tunnel-edge where the nodeName of the node tunnel . gRPC is disconnected, tunnel-cloud will delete the mapping of related podIp and node name
• tunnel-edge will use the edge node name and token to construct gRPC connection, and tunnel-cloud will pass authentication information analysis gRPC connect to the corresponding edge node, and construct a wrappedServerStream for each edge node to process (The same tunnel-cloud can handle multiple tunnel-edge connections)
• tunnel-cloud every one minute (considering the time for configmap to synchronize the pod mount file of tunnel-cloud tunnel-coredns hosts plug-in configuration file corresponding to the configmap synchronization edge node name and tunnel- Cloud podIp mapping; In addition, the introduction of configmap local mount file optimizes the performance of hosting mode when many clusters simultaneously synchronize tunnel-coredns
• tunnel-edge will send tunnel-cloud a normal heartbeat StreamMsg tunnel-cloud will respond after receiving the heartbeat (the heartbeat is to detect gRPC Stream Stream Is it normal?)
• StreamMsg includes heartbeat, TCP proxy and HTTPS request and other different types of messages; at the same time tunnel-cloud distinguishes with different edge nodes gRPC connection tunnel through context.node

HTTPS proxy

HTTPS module is responsible for establishing the HTTPS proxy of the cloud side, and forwarding the https request of the cloud component (for example: kube-apiserver) to the side service (for example: kubelet)

func (https *Https) Start(mode string) {    context.GetContext().RegisterHandler(util.CONNECTING, util.HTTPS, httpsmsg.ConnectingHandler)    context.GetContext().RegisterHandler(util.CONNECTED, util.HTTPS, httpsmsg.ConnectedAndTransmission)    context.GetContext().RegisterHandler(util.CLOSED, util.HTTPS, httpsmsg.ConnectedAndTransmission)    context.GetContext().RegisterHandler(util.TRANSNMISSION, util.HTTPS, httpsmsg.ConnectedAndTransmission)    if mode == util.CLOUD {        go httpsmng.StartServer()    }}

The Start function first registers the processing function of StreamMsg, and the CLOSED processing function mainly processes the message of closing the connection and starts the HTTPS Server.
When the cloud component sends a HTTPS tunnel-cloud , the serverHandler will first parse the node name from the request.Host field. If the TLS HTTP request object is written in the connection. When request.Host is not set, you need to parse the node name from request.TLS.ServerName. HTTPS Server reads request.Body and request.Header to construct an HttpsMsg structure, serializes it and encapsulates it into StreamMsg, sends StreamMsg through Send2Node and puts it in the Channel of the node corresponding to sends it to 160aca7813492c tunnel-edge

func (serverHandler *ServerHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {    var nodeName string    nodeinfo := strings.Split(request.Host, ":")    if context.GetContext().NodeIsExist(nodeinfo[0]) {        nodeName = nodeinfo[0]    } else {        nodeName = request.TLS.ServerName    }    ...    node.Send2Node(StreamMsg）}

tunnel-edge receives StreamMsg and calls ConnectingHandler function to process:

func ConnectingHandler(msg *proto.StreamMsg) error {    go httpsmng.Request(msg)    return nil}func Request(msg *proto.StreamMsg) {    httpConn, err := getHttpConn(msg)    ...    rawResponse := bytes.NewBuffer(make([]byte, 0, util.MaxResponseSize))    rawResponse.Reset()    respReader := bufio.NewReader(io.TeeReader(httpConn, rawResponse))    resp, err := http.ReadResponse(respReader, nil)    ...    node.BindNode(msg.Topic)    ...    if resp.StatusCode != http.StatusSwitchingProtocols {        handleClientHttp(resp, rawResponse, httpConn, msg, node, conn)    } else {        handleClientSwitchingProtocols(httpConn, rawResponse, msg, node, conn)    }}

ConnectingHandler will call Request to process the StreamMsg. TLS connection with the edge node Server through getHttpConn. Parse the data returned in the TLS HTTP Response, the Status Code is 200, and the response content is sent to tunnel-cloud , and the Status Code is 101. The binary data of the response TLS To tunnel-cloud , where StreamMsg.Type is CONNECTED.

tunnel-cloud receives the StreamMsg, it will call ConnectedAndTransmission for processing:

func ConnectedAndTransmission(msg *proto.StreamMsg) error {    conn := context.GetContext().GetConn(msg.Topic)    ...    conn.Send2Conn(msg)    return nil}

Obtain conn through msg.Topic(conn uid), and stuff the message into the pipe corresponding to the conn through Send2Conn

After the cloud HTTPS Server receives the CONNECTED message from the cloud, it believes that the HTTPS proxy is successfully established. And continue to execute handleClientHttp or handleClientSwitchingProtocols for data transmission. Here only analyze the data transmission process under handleClientHttp non-protocol promotion. The processing logic of HTTPS Client is as follows:

func handleClientHttp(resp *http.Response, rawResponse *bytes.Buffer, httpConn net.Conn, msg *proto.StreamMsg, node context.Node, conn context.Conn) {    ...    go func(read chan *proto.StreamMsg, response *http.Response, buf *bytes.Buffer, stopRead chan struct{}) {        rrunning := true        for rrunning {            bbody := make([]byte, util.MaxResponseSize)            n, err := response.Body.Read(bbody)            respMsg := &proto.StreamMsg{                Node:     msg.Node,                Category: msg.Category,                Type:     util.CONNECTED,                Topic:    msg.Topic,                Data:     bbody[:n],            }            ...            read <- respMsg        }        ...    }(readCh, resp, rawResponse, stop)    running := true    for running {        select {        case cloudMsg := <-conn.ConnRecv():            ...        case respMsg := <-readCh:            ...            node.Send2Node(respMsg)            ...        }    }    ...}

Here handleClientHttp will always try to read the data packets from the side-end component, and construct a StreamMsg of type TRANSNMISSION and send it to tunnel-cloud , tunnel-cloud after receiving StreamMsg, call ConnectedAndTransmission function and put StreamMsg into StreamMsg. Type corresponds to the HTTPS module in the conn.Channel

func handleServerHttp(rmsg *HttpsMsg, writer http.ResponseWriter, request *http.Request, node context.Node, conn context.Conn) {    for k, v := range rmsg.Header {        writer.Header().Add(k, v)    }    flusher, ok := writer.(http.Flusher)    if ok {        running := true        for running {            select {            case <-request.Context().Done():                ...            case msg := <-conn.ConnRecv():                ...                _, err := writer.Write(msg.Data)                flusher.Flush()                ...            }        }    ...}

After handleServerHttp receives StreamMsg, it will send msg.Data, which is the data packet of the side component, to the cloud component. The entire data flow is one-way sent from the edge to the cloud, as shown below:

For kubectl exec , the data flow is bidirectional. At this time, the kubelet will return a return packet with a StatusCode of 101, indicating that the protocol has been upgraded. Then tunnel-cloud and tunnel-edge will be cut to respectively. handleServerSwitchingProtocols and handleClientSwitchingProtocols HTTPS to complete the two-way transmission of the data stream.

The architecture is as follows:

The summary of the HTTPS module is as follows:

summary

• HTTPS: Responsible for establishing the cloud side HTTPS proxy (eg: cloud kube-apiserver <-> side kubelet) and transmitting data
• The role is similar to the TCP proxy, the difference is that the tunnel-cloud will read the edge node name carried in the HTTPS HTTPS proxy with the edge node; instead of the TCP agent randomly selects a cloud side tunnel for forwarding
• When cloud apiserver or other cloud applications access kubelet or other applications on edge nodes, tunnel-dns is hijacked by DNS (resolving the node name in Request host to tunnel-cloud podIp) and forwards the request to tunnel On the pod of tunnel-cloud encapsulates the request information as StreamMsg and sends it to tunnel-edge through the cloud edge tunnel corresponding to the node name, and tunnel-edge through the received StreamMsg field and the configured Addr field The certificate in the file establishes a TLS connection with the edge server, and writes the request information in into the 160aca78134b17 TLS connection. Tunnel-Edge from the TLS read Server connection to the edge of the end of the return data, encapsulates it into StreamMsg sent to Tunnel-Cloud , Tunnel-Cloud received write data Drive assembly Tunnel -Cloud establishing a connection.

TCP

TCP TCP proxy tunnel between the cloud control cluster and the edge independent cluster in the multi-cluster management:

func (tcp *TcpProxy) Start(mode string) {    context.GetContext().RegisterHandler(util.TCP_BACKEND, tcp.Name(), tcpmsg.BackendHandler)    context.GetContext().RegisterHandler(util.TCP_FRONTEND, tcp.Name(), tcpmsg.FrontendHandler)    context.GetContext().RegisterHandler(util.CLOSED, tcp.Name(), tcpmsg.ControlHandler)    if mode == util.CLOUD {    ...        for front, backend := range Tcp.Addr {            go func(front, backend string) {                ln, err := net.Listen("tcp", front)                ...                for {                    rawConn, err := ln.Accept()                    ....                    fp := tcpmng.NewTcpConn(uuid, backend, node)                    fp.Conn = rawConn                    fp.Type = util.TCP_FRONTEND                    go fp.Write()                    go fp.Read()                }            }(front, backend)        }    }

The Start function first registers the StreamMsg processing function, and the CLOSED processing function mainly processes the message of closing the connection, and then starts the TCP Server in the cloud.
After receiving the request from the cloud component, TCP Server will encapsulate the request into StremMsg and send it to StreamServer, which is sent by StreamServer to tunnel-edge , where StreamMsg.Type=FrontendHandler, StreamMsg.Node is from the node of the established cloud edge tunnel Choose one randomly.
tunnel-edge After receiving the StreamMsg, it will call the FrontendHandler function to process

func FrontendHandler(msg *proto.StreamMsg) error {    c := context.GetContext().GetConn(msg.Topic)    if c != nil {        c.Send2Conn(msg)        return nil    }    tp := tcpmng.NewTcpConn(msg.Topic, msg.Addr, msg.Node)    tp.Type = util.TCP_BACKEND    tp.C.Send2Conn(msg)    tcpAddr, err := net.ResolveTCPAddr("tcp", tp.Addr)    if err != nil {    ...    conn, err := net.DialTCP("tcp", nil, tcpAddr)    ...    tp.Conn = conn    go tp.Read()    go tp.Write()    return nil}

FrontendHandler first uses StreamMsg.Addr to establish a TCP connection with the Edge Server, starts the coroutine to asynchronously read and write TCP connections, and creates a new conn object (conn.uid=StreamMsg.Topic), and writes eammsg.Data to the TCP connection. tunnel-edge receives the returned data from Edge Server and encapsulates it as StreamMsg (StreamMsg.Topic=BackendHandler) and sends it to tunnel-cloud

The whole process is shown in the figure:

summary

• TCP proxy between the cloud and the edge in the multi-cluster management
• The cloud component accesses the server on the edge TCP TCP server in the cloud will encapsulate the request into StreamMsg after receiving the request. Pass the cloud side tunnel (choose one randomly from the connected tunnels, so it is recommended that there is only one tunnel - eDGE used in the scene under the TCP agent) to Tunnel-edge , * edge Tunnel- end Server receives Addr field is established by the edge of StreamMag the TCP connection, and requests to write the TCP connection. tunnel-edge reads the return message of the edge server from the TCP connection, and sends it to the tunnel-cloud , tunnel-cloud and writes it to the cloud component and established by 160aca78134cc0 Server

Outlook

• Support more network protocols (has supported HTTPS and TCP )
• Support cloud access to edge node business pod server
• When multiple edge nodes join the cluster at the same time, the multi-copy tunnel-cloud pod tunnel-coredns hosts plug-in configuration file corresponds to configmap. Although the probability is low, theoretically there is still the possibility of write conflicts. Sex

Refs

• kubernetes-reading-notes

[Tencent Cloud Native] Yunshuo new products, Yunyan new technology, Yunyou Xinhuo, Yunxiang information, scan the QR code to follow the public account of the same name, and get more dry goods in time! !