Take you through Jetty custom ProxyServlet to achieve reverse proxy service

Abstract: has recently done a task of mapping a certain component UI in K8s to a link address of a custom rule through a reverse proxy to provide users with access. So by the way, I studied Jetty's ProxyServlet.

This article is shared from Huawei Cloud Community " Jetty Custom ProxyServlet to Realize Reverse Proxy Service (including source code analysis) ", author: Xiao Yan.

1. Background overview

Recently, there is a need to map a certain component UI in K8s to a link address of a custom rule through a reverse proxy to provide users with access. So by the way, I studied Jetty's ProxyServlet. Do an analysis here, if you have a poor understanding, I hope you can add corrections.

Second, the basic architecture of Jetty

Jetty is a Servlet engine, its architecture is relatively simple, it is also a scalable and very flexible application server, it has a basic data model, this data model is Handler, all components that can be extended can be used as a Handler, add In Server, Jetty is to help you manage these Handlers.

The core components of Jetty are composed of two components: Server and Connector. The entire Server component is based on the Handler container. Another indispensable component in Jetty is Connector, which is responsible for accepting client connection requests and assigning requests to one Process the queue to execute. There are some dispensable components in Jetty, and we can expand on it. Like JMX, we can define some MBeans and add them to the Server. When the Server starts, these Beans will work together.

The core of Jetty is built around the Server class. The Server class inherits the Handler and associates the Connector and the Container. Container is the container for managing MBeans. The extension of Jetty's Server is mainly to implement Handlers one by one and add Handlers to the Server. The Server provides access rules for calling these Handlers. The life cycle management of all components of Jetty is based on the observer template design to realize LifeCycle.

Three, Handler's architecture

Jetty is mainly designed based on Handler. The architecture of Handler affects all aspects of Jetty. The following summarizes the types and functions of Handler:

Jetty mainly provides two Handler types. One is HandlerWrapper, which can delegate a Handler to another class to execute. If we want to add a Handler to Jetty, then we must delegate this Handler to the Server to call. With the ScopeHandler class, we can intercept the execution of the Handler. Before or after calling the Handler, we can do some other things, similar to Valve in Tomcat; the other Handler type is HandlerCollection, this Handler class can assemble multiple Handlers together, A Handler chain is formed to facilitate our expansion.

Four, coding design

Here I provide a design framework, the specific content can be customized according to needs.

public class RestApi {
    private static final Logging LOGGER = Logging.getLogging(RestApi.class.getName());
​
    private Server server;
​
    /**
     * 启动方法，需要在程序启动时调用该方法
     */
    public void start() {
        try {
            ContextHandlerCollection collection = new ContextHandlerCollection();
            WebAppContext appContext = new WebAppContext();
            appContext.setContextPath("/");
            // 设置资源文件地址，可略
            appContext.setResourceBase("/opt/appHome/myDemo/webapp");
            // 设置web.xml，可在里面进行一些Servlet配置，可略
            appContext.setDescriptor("/opt/appHome/myDemo/webapp/WEB-INF/web.xml"); 
            appContext.setParentLoaderPriority(true);
            collection.addHandler(appContext);
            addProxyHandler(collection);
​
            server = new Server(8080);
            server.setHandler(collection);
            server.start();
        } catch (Throwable t) {
            LOGGER.error("Start RESTful API server failed", t);
        }
    }
 
    private static void addProxyHandler(ContextHandlerCollection collection) {
        ProxyServlet proxyServlet = new WebProxyServlet();      // 添加自定义ProxyServlet
​
        ServletHolder holder = new ServletHolder(proxyServlet);
​
        holder.setInitParameter("idleTimeout", 120000);         // 设置空闲释放时间
        holder.setInitParameter("timeout", 300000);             // 设置超时时间
        holder.setInitParameter("maxConnections", 256);         // 设置最大连接数
​
        ServletContextHandler contextHandler = new ServletContextHandler();
        contextHandler.addServlet(holder, "/proxy/*");
        contextHandler.setContextPath("/demo");
        collection.addHandler(contextHandler);
    }
}

Customize ProxyServlet, here is a list of some commonly used rewriting methods, there are many ways to query the document to rewrite by yourself

public class WebProxyServlet extends ProxyServlet {
    private static final Logging LOGGING = Logging.getLogging(WebProxyServlet.class);
​
    /**
     * 自定义目标地址重写方法
     */
    @Override
    protected String rewriteTarget(HttpServletRequest request) { }      
 
    /**
     * 自定义重写错误处理方法
     */
    @Override
    protected void onProxyRewriteFailed(HttpServletRequest clientRequest, HttpServletResponse clientResponse) { }
 
    /**
     * 自定义response错误处理方法
     */
    @Override
    protected void onProxyResponseFailure(
        HttpServletRequest clientRequest,
        HttpServletResponse proxyResponse,
        Response serverResponse,
        Throwable failure) { }
 
    /**
     * 自定义response头filter
     */
    @Override
    protected String filterServerResponseHeader(
        HttpServletRequest clientRequest,
        Response serverResponse,
        String headerName,
        String headerValue) { }
 
    /**
     * 自定义头XForwarded设置
     */
    @Override
    protected void addXForwardedHeaders(HttpServletRequest clientRequest, Request proxyRequest) { }
}

Five, request processing process

The following is a preliminary sorting out by tracking the source code, from the request request to the entire process of returning the response

Six, source code analysis

1. Request forwarding part

When Jetty receives a request, Jetty passes the request to the ContextHandlerCollection registered in the Server for execution, and check the source code of the handle method of the Service

public void handle(HttpChannel channel) throws IOException, ServletException {
      String target = channel.getRequest().getPathInfo();
      Request request = channel.getRequest();
      Response response = channel.getResponse();
      if (LOG.isDebugEnabled()) {
          LOG.debug("{} {} {} on {}", new Object[]{request.getDispatcherType(), request.getMethod(), target, channel});
      }
​
      if (!HttpMethod.OPTIONS.is(request.getMethod()) && !"*".equals(target)) {
          this.handle(target, request, request, response);
      } else if (!HttpMethod.OPTIONS.is(request.getMethod())) {
          request.setHandled(true);
          response.sendError(400);
      } else {
          this.handleOptions(request, response);
          if (!request.isHandled()) {
              this.handle(target, request, request, response);
          }
      }
​
      if (LOG.isDebugEnabled()) {
          LOG.debug("handled={} async={} committed={} on {}", new Object[]{request.isHandled(), request.isAsyncStarted(), 
                                                                           response.isCommitted(), channel});
      }
  }

The this.handle(target, request, request, response) method called here is actually the handle method of the parent class HandlerWrapper

 public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
     throws IOException, ServletException {
     Handler handler = this._handler;
     if (handler != null) {
         handler.handle(target, baseRequest, request, response);
     }
 }

Server.setHandler(collection) was called when the server was created, so the handle method of ContextHandlerCollection is called here

public void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
    throws IOException, ServletException {
    ContextHandlerCollection.Mapping mapping = (ContextHandlerCollection.Mapping)this._handlers.get();
    if (mapping != null) {
        Handler[] handlers = mapping.getHandlers();
        if (handlers != null && handlers.length != 0) {
            if (handlers.length == 1) {
                handlers[0].handle(target, baseRequest, request, response);
            } else {
                HttpChannelState async = baseRequest.getHttpChannelState();
                if (async.isAsync()) {
                    ContextHandler context = async.getContextHandler();
                    if (context != null) {
                        Handler branch = (Handler)mapping._contextBranches.get(context);
                        if (branch == null) {
                            context.handle(target, baseRequest, request, response);
                        } else {
                            branch.handle(target, baseRequest, request, response);
                        }
​
                        return;
                    }
                }
​
                int limit;
                if (target.startsWith("/")) {
                    Trie<Entry<String, ContextHandlerCollection.Branch[]>> pathBranches = mapping._pathBranches;
                    if (pathBranches == null) {
                        return;
                    }
​
                    int l;
                    for(limit = target.length() - 1; limit >= 0; limit = l - 2) {
                        Entry<String, ContextHandlerCollection.Branch[]> branches = 
                            (Entry)pathBranches.getBest(target, 1, limit);
                        if (branches == null) {
                            break;
                        }
​
                        l = ((String)branches.getKey()).length();
                        if (l == 1 || target.length() == l || target.charAt(l) == '/') {
                            ContextHandlerCollection.Branch[] var12 = 
                                (ContextHandlerCollection.Branch[])branches.getValue();
                            int var13 = var12.length;
​
                            for(int var14 = 0; var14 < var13; ++var14) {
                                ContextHandlerCollection.Branch branch = var12[var14];
                                branch.getHandler().handle(target, baseRequest, request, response);
                                if (baseRequest.isHandled()) {
                                    return;
                                }
                            }
                        }
                    }
                } else {
                    Handler[] var17 = handlers;
                    limit = handlers.length;
​
                    for(int var19 = 0; var19 < limit; ++var19) {
                        Handler handler = var17[var19];
                        handler.handle(target, baseRequest, request, response);
                        if (baseRequest.isHandled()) {
                            return;
                        }
                    }
                }
            }
        }
    }
}

From the source code above, we can see that the handle method of ContextHandlerCollection continues to call the handle method of collection.addHandler set in ServletContextHandler. Through tracking, we can find that the handle of the parent class ScopedHandler is actually called here --> doScope --> nextScope

 public final void handle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
     throws IOException, ServletException {
     if (this.isStarted()) {
         if (this._outerScope == null) {
             this.doScope(target, baseRequest, request, response);
         } else {
             this.doHandle(target, baseRequest, request, response);
         }
     }
 }
​
 public void doScope(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
     throws IOException, ServletException {
     this.nextScope(target, baseRequest, request, response);
 }
​
 public final void nextScope(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
     throws IOException, ServletException {
     if (this._nextScope != null) {
         this._nextScope.doScope(target, baseRequest, request, response);
     } else if (this._outerScope != null) {
         this._outerScope.doHandle(target, baseRequest, request, response);
     } else {
         this.doHandle(target, baseRequest, request, response);
     }
 }

Looking at ServletContextHandler, you can find that the following three handlers are mainly registered, all of which are subclasses of ScopedHandler, that is, this._nextScope in the nextScope method

protected SessionHandler _sessionHandler;
protected SecurityHandler _securityHandler;
protected ServletHandler _servletHandler;

SessionHandler is a layer of packaging (decorator mode) for ServletHandler, which is used for preprocessing of some sessions, and SecurityHandler does some security-related analysis from the name. These two specifics will not be analyzed, and we will look at ServletHandler directly. DoScope method

public void doScope(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
    throws IOException, ServletException {
    String old_servlet_path = baseRequest.getServletPath();
    String old_path_info = baseRequest.getPathInfo();
    DispatcherType type = baseRequest.getDispatcherType();
    ServletHolder servletHolder = null;
    Scope oldScope = null;
    MappedResource<ServletHolder> mapping = this.getMappedServlet(target);
    if (mapping != null) {
        servletHolder = (ServletHolder)mapping.getResource();
        if (mapping.getPathSpec() != null) {
            PathSpec pathSpec = mapping.getPathSpec();
            String servletPath = pathSpec.getPathMatch(target);
            String pathInfo = pathSpec.getPathInfo(target);
            if (DispatcherType.INCLUDE.equals(type)) {
                baseRequest.setAttribute("javax.servlet.include.servlet_path", servletPath);
                baseRequest.setAttribute("javax.servlet.include.path_info", pathInfo);
            } else {
                baseRequest.setServletPath(servletPath);
                baseRequest.setPathInfo(pathInfo);
            }
        }
    }
​
    if (LOG.isDebugEnabled()) {
        LOG.debug("servlet {}|{}|{} -> {}", 
                  new Object[]{baseRequest.getContextPath(), 
                               baseRequest.getServletPath(), 
                               baseRequest.getPathInfo(), 
                               servletHolder});
    }
​
    try {
        oldScope = baseRequest.getUserIdentityScope();
        baseRequest.setUserIdentityScope(servletHolder);
        this.nextScope(target, baseRequest, request, response);
    } finally {
        if (oldScope != null) {
            baseRequest.setUserIdentityScope(oldScope);
        }
​
        if (!DispatcherType.INCLUDE.equals(type)) {
            baseRequest.setServletPath(old_servlet_path);
            baseRequest.setPathInfo(old_path_info);
        }
    }
}

Here are some settings for baseRequest, add the registered ServletHolder set into baseRequest, and then continue to call this.nextScope(target, baseRequest, request, response), according to the nextScope method above, after all scopes are executed, doHandle is executed Method, continue to skip SessionHandler and SecurityHandler, look at the doHandle method of ServletHandler

public void doHandle(String target, Request baseRequest, HttpServletRequest request, HttpServletResponse response) 
    throws IOException, ServletException {
    ServletHolder servletHolder = (ServletHolder)baseRequest.getUserIdentityScope();
    FilterChain chain = null;
    if (servletHolder != null && this._filterMappings != null && this._filterMappings.length > 0) {
        chain = this.getFilterChain(baseRequest, target.startsWith("/") ? target : null, servletHolder);
    }
​
    if (LOG.isDebugEnabled()) {
        LOG.debug("chain={}", new Object[]{chain});
    }
​
    try {
        if (servletHolder == null) {
            this.notFound(baseRequest, request, response);
        } else {
            ServletRequest req = request;
            if (request instanceof ServletRequestHttpWrapper) {
                req = ((ServletRequestHttpWrapper)request).getRequest();
            }
​
            ServletResponse res = response;
            if (response instanceof ServletResponseHttpWrapper) {
                res = ((ServletResponseHttpWrapper)response).getResponse();
            }
​
            servletHolder.prepare(baseRequest, (ServletRequest)req, (ServletResponse)res);
            if (chain != null) {
                chain.doFilter((ServletRequest)req, (ServletResponse)res);
            } else {
                servletHolder.handle(baseRequest, (ServletRequest)req, (ServletResponse)res);
            }
        }
    } finally {
        if (servletHolder != null) {
            baseRequest.setHandled(true);
        }
    }
}

doHandle is mainly to take out the registered FilterChain ServletHolder, if there is a Filter, execute the chain.doFilter method first, otherwise execute the servletHolder.handle. I did not set the filter, so just look at the handle method of ServletHolder.

public void handle(Request baseRequest, ServletRequest request, ServletResponse response) 
      throws ServletException, UnavailableException, IOException {
      try {
          Servlet servlet = this.getServletInstance();
          if (servlet == null) {
              throw new UnavailableException("Servlet Not Initialized");
          }
​
          servlet.service(request, response);
      } catch (UnavailableException var5) {
          this.makeUnavailable(var5).service(request, response);
      }
  }

The service method of Servlet in ServletHolder is called here, which means we have reached our custom class WebProxyServlet. Because there is no rewriting, the service method of ProxyServlet is called here.

protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
      int requestId = this.getRequestId(request);
      String rewrittenTarget = this.rewriteTarget(request);
      if (this._log.isDebugEnabled()) {
          StringBuffer uri = request.getRequestURL();
          if (request.getQueryString() != null) {
              uri.append("?").append(request.getQueryString());
          }
​
          if (this._log.isDebugEnabled()) {
              this._log.debug("{} rewriting: {} -> {}", new Object[]{requestId, uri, rewrittenTarget});
          }
      }
​
      if (rewrittenTarget == null) {
          this.onProxyRewriteFailed(request, response);
      } else {
          Request proxyRequest = this.newProxyRequest(request, rewrittenTarget);
          this.copyRequestHeaders(request, proxyRequest);
          this.addProxyHeaders(request, proxyRequest);
          AsyncContext asyncContext = request.startAsync();
          asyncContext.setTimeout(0L);
          proxyRequest.timeout(this.getTimeout(), TimeUnit.MILLISECONDS);
          if (this.hasContent(request)) {
              if (this.expects100Continue(request)) {
                  DeferredContentProvider deferred = new DeferredContentProvider(new ByteBuffer[0]);
                  proxyRequest.content(deferred);
                  proxyRequest.attribute(CONTINUE_ACTION_ATTRIBUTE, () -> {
                      try {
                          ContentProvider provider = this.proxyRequestContent(request, response, proxyRequest);
                          (new ProxyServlet.DelegatingContentProvider(request, proxyRequest, 
                                                                      response, provider, deferred)).iterate();
                      } catch (Throwable var6) {
                          this.onClientRequestFailure(request, proxyRequest, response, var6);
                      }
​
                  });
              } else {
                  proxyRequest.content(this.proxyRequestContent(request, response, proxyRequest));
              }
          }
          this.sendProxyRequest(request, response, proxyRequest);
      }

}

So far we have called the most critical method rewriteTarget that we rewrite. This method can customize the logic to resolve the address of the request and return the target address to be proxied to. Use the target address to form a proxyRequest and finally call sendProxyRequest to implement proxy forwarding.

2. Response receiving part

If you continue to follow sendProxyRequest, you will see that a ProxyResponseListener has been created. I won’t track it in detail here. I will mainly talk about the process. If you are interested, you can take a look at it yourself. Response returns will trigger the onHeader method through the reflection mechanism. ProxyServlet overrides this method and jumps to the onServerResponseHeaders method.

 public void onHeaders(Response proxyResponse) {
     ProxyServlet.this.onServerResponseHeaders(this.request, this.response, proxyResponse);
 }

This method is to set the header content of the Response. One of them is to get the headerValue and call the this.filterServerResponseHeader method. We can also customize the headerValue of the return body by overriding this method.

Seven, summary

At this point, Jetty's ProxyServlet operating principle and custom methods are roughly sorted out. There are many things that are missed and not well understood. I hope everyone can make corrections. It is very meaningful to take a moment to read the source code occasionally at work, which can improve the understanding of the technology used, and also learn to appreciate the clever design of these frameworks.

Click to follow and learn about Huawei Cloud's fresh technology for the first time~