Summary of using nginx as a reverse proxy for grpc

This article records the pits and solutions of using nginx as a reverse proxy for gRPC.

background

As we all know, nginx is a high-performance web server, often used for load balancing and reverse proxy . The so-called reverse proxy corresponds to the forward proxy. The forward proxy is the "proxy" we understand in the conventional sense: for example, it is impossible to access Google in China under normal circumstances. If we need to access, we need to go through a layer of proxy. Forward. This forward proxy proxy is the server (that is, google), while the reverse proxy is the opposite. The proxy is the client (that is, the user). After the user’s request reaches nginx, nginx will proxy the user’s request to the actual back The end service initiates a request and returns the result to the user.

(Picture from Wikipedia)

The forward proxy and reverse proxy are actually defined from the perspective of the user. The forward proxy is the service requested by the proxy user, and the reverse proxy is the proxy user initiates a request to the service. A very important difference between the two:

The forward proxy server does not perceive the requester, and the reverse proxy requester does not perceive the server.

Consider the above example. When you access Google through a proxy, Google can only perceive that the request comes from the proxy server, but cannot directly perceive you (of course, it can also be traced through cookies and other means); and when you access google through nginx reverse proxy, you It is not aware of which back-end server the request is forwarded to.

The most common scenario where nginx is used for reverse proxy is the http protocol we are familiar with. You can easily define a reverse proxy rule by configuring the nginx.conf file:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       80;
        server_name  localhost;

        
        location / {
            proxy_pass http://domain;
        }
    }
}

Nginx has supported the reverse proxy of the gRPC protocol since 1.13.10, and the configuration is similar:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       81 http2;
        server_name  localhost;

        
        location / {
            grpc_pass http://ip;
        }
    }
}

But when the demand scenario is more complicated, it turns out that nginx's gRPC module actually has many pits, and the realization ability is not as complete as http. Problems will arise when the http solution is applied.

Scenes

At the beginning, our scenario was very simple. A simple C/S architecture was implemented through the gRPC protocol:

However, this simple direct connection is not feasible in some scenarios. For example, if the client and server are not connected to each other in two network environments, it is impossible to access the service through a simple gRPC connection. One solution is to forward through the intermediate proxy server, using the nginx reverse proxy gRPC method mentioned above:

The nginx proxy is deployed on a cluster that can be accessed by both environments, thus realizing gRPC access across network environments. The question that follows is how to configure this routing rule? Note that the target nodes of our initial gRPC are clear, that is, the ip addresses of server1 and server2. When a layer of nginx proxy is added in between, the objects of gRPC requests initiated by the client are the ip addresses of nginx proxy. After the client establishes a connection with nginx, how does nginx know whether to forward the request to server1 or server2? (Here server1 and server2 are not simply redundant deployments of the same service. It may be necessary to determine who responds according to the attributes of the request, such as user id, etc., so load balancing cannot be used to randomly select a response request)

Solution

If it is the http protocol, there are many ways to implement it:

• Distinguish by path

Request to add the server information to the path, for example: /server1/service/method , and then restore the original request when nginx forwards the request:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       80;
        server_name  localhost;

        location ~ ^/server1/ {
            proxy_pass http://domain1/;
        }
        
        location ~ ^/server2/ {
            proxy_pass http://domain2/;
        }
    }
}

Pay attention to http://domain/ . If there is no such slash, the requested path will be /server1/service/method , and the server can only respond to /service/method , so it will report a 404 error.

• Distinguish by request parameters

You can also put server1's information in the request parameters:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       80;
        server_name  localhost;

        location /service/method {
            if ($query_string ~ x_server=(.*)) {
                proxy_pass http://$1;
            }
        }
    }
}

But for gRPC, it's not that simple. First, gRPC does not support URI writing. Requests forwarded by nginx will retain the original path and cannot be modified during forwarding. This means that the first method above is not feasible. Secondly, gRPC is based on HTTP 2.0 protocol . HTTP2 does not have the concept of queryString. There is a :path request header representing the path of the request, such as /service/method , and this path cannot carry request parameters, that is, :path cannot be written as /service/method?server=server1 . This means that the second method described above is also not feasible.

Note that the request header :path specifies the path of the request. Then, can we directly modify :path :

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       80 http2;
        server_name  localhost;

        location ~ ^/(.*)/service/.* {
            grpc_set_header :path /service/$2;
            grpc_pass http://$1;
        }
    }
}

However, actual verification shows that this method is not feasible. Modifying :path directly will cause the server to report an error. One possible error is as follows:

rpc error: code = Unavailable desc = Bad Gateway: HTTP status code 502; transport: received the unexpected content-type "text/html"

After capturing the packet, it was found that grpc_set_header did not cover :path , but added a new request header, which is equivalent to two :path request header. It may be that the server reported a 502 error for this reason.

When the mountain is exhausted, I think of the metadata function of gRPC. We can store the server information in the metadata on the client side, and then forward it to the corresponding back-end service according to the server information in the metadata during nginx routing, thus achieving our demand. For the go language, setting metadata needs to implement the PerRPCCredentials interface, and then pass in an instance of this implementation class when initiating a connection:

type extraMetadata struct {
    Ip string
}

func (c extraMetadata) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
    return map[string]string{
        "x-ip": c.Ip,
    }, nil
}

func (c extraMetadata) RequireTransportSecurity() bool {
    return false
}

func main(){
    ...
    // nginxProxy是nginx proxy的ip或域名地址
    var nginxProxy string
    // serverIp是根据请求属性计算好的后端服务的ip
    var serverIp string
    con, err := grpc.Dial(nginxProxy, grpc.WithInsecure(),
        grpc.WithPerRPCCredentials(extraMetadata{Ip: serverIp}))
}

Then forward to the corresponding server according to this metadata in the nginx configuration:

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    server {
        listen       80 http2;
        server_name  localhost;

        location ~ ^/service/.* {
            grpc_pass grpc://$http_x_ip:8200;
        }
    }
}

Note that the syntax $http_x_ip used here refers to the metadata information x-ip This method is valid, and the client can successfully access the gRPC service of the server through the nginx proxy.

to sum up

There are too few documents for the gRPC module of nginx. The official document only gives a few instructions for the purpose, and does not explain the metadata method. The online documents are rarely involved, which leads to two or three days of investigation. Summarize the whole process here, hoping to help people who encounter the same problem.