New feature of SuperEdge cloud edge tunnel: SSH operation and maintenance of edge nodes from the cloud

background

In the scenario of edge clusters, edge nodes are distributed in different areas, and there is a one-way network between the edge nodes and the cloud. The edge nodes can access the cloud nodes, and the cloud nodes cannot directly access the edge nodes, which brings great difficulties to the operation and maintenance of the edge nodes. Great inconvenience. If you can log in to the edge node from the cloud through SSH, it can simplify the operation and maintenance of the node. In response to this demand, the SuperEdge project expanded the capabilities of the tunnel component and added an SSH module, allowing users to log in to edge nodes from the cloud through SSH.

demand analysis

The nodes of the edge cluster are distributed in different areas. The edge nodes can access the cloud master node, but the master node cannot access the nodes in the edge area. The user wants to log in to the node through the port of the master node to log in to perform operation and maintenance work.

Conventional plan

Using SSH port forwarding technology can realize the SSH operation and maintenance edge node function, the specific content is shown in the following figure:

1. The edge nodes node-A and node-B bind the port-A and port-B ports of the cloud master-A node with the local port 22 (the port of the SSH Server) through SSH remote forwarding (ssh -R)
2. The user establishes an SSH tunnel with master-A through SSH dynamic forwarding (ssh -D) and simultaneously monitors the local-port port locally
3. Local-port requests will be transmitted to master-A via SSH connection and forwarded by master-A. For example, SSH login node-A: ssh -o ProxyCommand=”nc -X 5 -x 127.0.0.1:local-port 127.0.0.1 port-A” root@127.0.0.1, 127.0.0.1 port-A is the destination address requested by master-A when forwarding.

Disadvantages of the conventional scheme:

• Edge node mapping port management is complicated
  As shown in Figure 2, node-A and node-B map the local 22 port on master-A to different ports. SSH login to the target node needs to specify the port mapped on master-A, when there are many edge nodes Port mapping management is very complicated, which directly leads to the infeasibility of the solution.
• Multiple connections involved in the request increase the probability of errors
  Take SSH login node-A as an example, as shown in Figure 1, sshClient->local-port, user->master-A, master-A->port-A, master-A->node-A, node-A ->sshServer, a total of 5 connections need to be established.
• Trouble maintenance of SSH on the cloud and on the side
  The SSH connection between the edge node and the cloud node needs to be established on the edge node, and the connection does not have the ability to disconnect and reconnect, which is troublesome to maintain.

tunnel scheme

Architecture design

1. The SSH Client requests the tunnel-cloud service to leak to the NodePort-1 port of the external network, and the service forwards the request to tunnel-cloud pod-A or tunnel-cloud pod-B according to the load balancing policy.
2. If the request is forwarded to pod-A (R.1.1), since node-A does not establish a tunnel with pod-A, pod-A will query coredns to obtain the podIp of pod-B that establishes a tunnel with node-A, and then forward the request to pod-B (R.1.2)
3. After pod-B receives the request, it forwards the request to the tunnel-edge of node-A through the tunnel, and the tunnel-edge forwards the request to the SSH server.

Advantages of this program

• The tunnel between the cloud and the edge node does not map ports, avoiding port management.
• Reduce the number of connections established during the request process and reduce the probability of errors
• The cloud and edge tunnels have a disconnect and reconnect mechanism to reduce maintenance costs

The establishment of the cloud side tunnel

Use the gRPC open source project to build a long connection tunnel, and gRPC realizes the disconnection and reconnection mechanism.

1. tunnel-edge sends a request to establish a gRPC connection to tunnel-cloud
   tunnel-edge is sending a request to establish a connection to NodePort-2 of tunnel-cloud. The request carries the node name and token information of the node where it is located. The tunnel-cloud service forwards the request to the tunnel-cloud pod according to the load balancing policy, as shown in Figure 3. As shown, forward the request to tunnel-cloud pod-B
2. tunnel-cloud registers the podIp of this pod and the node name information of the node where the tunnel-edge is located with coredns
   tunnel-cloud verifies the token information in the tunnel-edge request information. After the verification is passed, the node name in the node request information and the podIp of the pod are written to coredns
3. tunnel-cloud returns a message that the gRPC connection is established successfully
4. Send custom protocol message StreamMsg between tunnel-edge and tunnel-cloud through gRPC long connection
   For the field definition of the custom protocol message, please refer to the to understand the SuperEdge cloud side tunnel

Implementation of tunnel-cloud forwarding

1. SSH Client requests the NodePort-1 port of tunnel-cloud service, and sends an http request with a method of Connect
   The method sent by the SSH Client through the tool netcat or crokscrew is the CONNECT HTTP request, that is, the content of the ProxyCommand (ProxyCommand = "nc -X connect -x tunnel-cloudIp:NodePort-1 node-A 22" ). The parameter definitions are as follows:

   • -X: The parameter is the type of protocol, the connect specified here is HTTP CONNECT
   • -x: the ip address and port of the HTTP Server, the tunnel-cloudIp:NodePort-1 specified here is the ip and port of the tunnel-cloud service exposed to the external network
   • node-A is the node name of the edge node
   • 22 is the listening port of the edge node SSH Server

   The tunnel-cloud service forwards the SSH Client request to the tunnel-cloud pod according to the load balancing policy. As shown in Figure 3 of the architecture design, if it is forwarded to tunnel-cloud pod-A, the HTTP server of tunnel-cloud will receive the message CONNECT node-A: 22 HTTP/1.0 , where node-A is the node name of the edge node, and 22 is the port monitored by the edge node SSH Server. Since node-A does not establish a cloud edge tunnel with tunnel-cloud pod-A, Therefore, the HTTP Server will request coredns to obtain the podIp of the tunnel-cloud corresponding to the node name of node-A, which is tunnel-cloud pod-B, and pod-A will forward the request to pod-B

2. tunnel-cloud sends custom protocol messages to tunnel-edge (StreamMsg.Type is connecting)
   tunnel-cloud will obtain the tunnel between the cloud and the edge node according to the HTTP CONNECT request information, and send a custom protocol message to tunnel-edge through the cloud edge tunnel to establish a TCP connection with the SSH Server, the type is connecting
3. tunnel-edge sends a message to establish a TCP connection to the SSH Server
   tunnel-edge sends a request to establish a TCP connection according to the ip and port of the SSH Server in the message after receiving a custom protocol message of the connecting type
4. SSH Server returns a successful TCP connection establishment message to tunnel-edge
5. Tunnel-edge returns a custom protocol message (StreamMsg.Type is conneted) to tunnel-cloud
   After tunnel-edge receives the message that the connection is established successfully, it sends tunnel-cloud a message of the type connected custom protocol that the TCP connection has been established.
6. tunnel-cloud returns a response message with SSH Client status 200
   After tunnel-cloud receives the connected custom protocol message, the SSH Client will return a message with a status code of 200: HTTP/1.1 200 Connection established
7. SSH Client sends SSH protocol messages to tunnel-cloud
   After receiving the response message with the status code of 200, the SSH Client uses the tunnel established with tunnel-cloud to send SSH protocol data
8. tunnel-cloud encapsulates the SSH protocol message as a custom protocol message (StreamMsg.Type is content) and sends it to tunel-edge
9. tunnel-edge sends SSH protocol messages to SSH Server through TCP connection
10. SSH Server sends SSH protocol messages to tunnel-edge
11. tunnel-edge encapsulates the SSH protocol message as a custom message of content type and sends it to tunnel-cloud
12. tunnel-cloud returns SSH protocol messages to SSH Client

SSH login node

function demonstration video link

Preconditions

Install corkscrew , Mac install directly with brew

brew install corkscrew

Or, install netcat (centos)

yum install -y netcat

SSH login node

To log in to the edge node node-A-1 through SSH, you can use the following command:

ssh -o ProxyCommand= "corkscrew masterIP  cloud-port node-A-1  22"  root@127.0.0.1

or

ssh -o ProxyCommand= "nc -X connect -x masterIP:cloud-port  node-A-1 22" root@127.0.0.1

• materIP: the external network ip of the node where the master node is located
• cloud-port: NodePort port, the server port of the corresponding SSH module
  Get cloud-port

kubectl -n edge-system get svc tunnel-cloud -o=jsonpath='{range .spec.ports[*]}{.name}{"\t"}{.nodePort}{"\n"}{end}' | grep ssh | awk '{print $2}'

to sum up

With this solution, users do not need to manually build, and can quickly SSH into the edge node to implement operation and maintenance work. At the same time, the tunnel provided by this solution is easier to maintain and improve stability than the traditional solution, and the user experience is improved.

Future outlook

Support the unified SSH operation and maintenance edge node from the cloud is an enhancement tunnel components, but also on article read SuperEdge cloud edge tunnel realize the vision of, and after SuperEdge the open source community small partners also tunnel assembly raised new demands, as follows :

• Support cloud apiserver to access edge webhook server
• Support cross-regional mutual visits between services

Cooperation and open source

The new features of cloud edge tunnel SSH operation and maintenance edge nodes have been SuperEdge release 0.4.0 , and everyone is welcome to experience it. to more complex edge network scenarios. Companies, organizations and individuals interested in edge computing are also welcome to jointly build the 160ed39fb147dc SuperEdge edge container project.

<img src="https://main.qcloudimg.com/raw/ded3f46433da0f1727caf3630effadf7.png" style="zoom:50%;" />

[Tencent Cloud Native] Yunshuo new products, Yunyan new technology, Yunyou Xinhuo, Yunxiang information, scan the QR code to follow the public account of the same name, and get more dry goods in time! !