Cloud native web service framework ESA Restlight

ESA Stack (Elastic Service Architecture) is a technology brand incubated by the OPPO Cloud Computing Center. It is committed to microservice-related technology stacks to help users quickly build high-performance, highly available cloud-native microservices. Products include high-performance Web service frameworks, RPC frameworks, service governance frameworks, registration centers, configuration centers, call chain tracking systems, Service Mesh, Serverless and other products and research directions.

Some current products have been open sourced

Open source main : 160e40f8c52d32 https://www.esastack.io/

Github: https://github.com/esastack

Restlight project address: https://github.com/esastack/esa-restlight

Restlight document address: https://www.esastack.io/esa-restlight/

All technical enthusiasts are welcome to join and discuss learning and progress together.

This article will inevitably mention Spring MVC many times, and does not mean to compete with it. Restlight is an independent web framework with its own persistence.

Current Status of Traditional Web Service Frameworks in the Java Industry

Spring MVC

When it comes to Web service frameworks, Spring MVC in the Java field is no one knows, no one knows. Realize request routing matching, filters, interceptors, serialization, deserialization, parameter binding, return value analysis and other capabilities on the basis of Tomcat (maybe Jetty, Undertow, etc.). Due to its rich functions and deep integration with the Spring container and Spring Boot, which have a huge number of users today, Spring MVC is almost the best choice for many companies' Web service frameworks.

Spring MVC in this article refers to the generalized Web service framework of Tomcat + Spring MVC

Resteasy

Resteasy is also a relatively mature Rest framework in the Java system. It is an open source project of JBoss. It fully implements the JAX-RS standard and helps users quickly build Rest services. It also provides a Resteasy JAX-RS client framework to facilitate users. Rest service call. Resteasy is integrated and used in many three-party frameworks, such as Dubbo, SOFA RPC and other well-known frameworks.

Is Spring MVC almighty?

In a sense, it is really omnipotent. Spring MVC almost has most of the capabilities that a traditional Web service should have. Whether it is a simple Rest service, an All In One console service, or an RPC service in Spring Cloud, Spring MVC can be used. .

However, with the evolution and changes of microservice technology, especially the prevailing concept of cloud-native microservices, this all-rounder seems to have some dissatisfaction.

performance

function and performance 160e40f8c52ef3

Spring MVC design is more of a function-oriented design. By viewing Spring's source code, you can see various high-level design patterns and interface designs, which makes Spring MVC an "all-rounder". However, complex design and functions also come at a price, that is, a compromise on performance, and sometimes some performance has to be given up for the sake of function or design.

Tomcat thread model

Spring MVC uses a single worker thread pool to process requests

We can use server.tomcat.threads.max to configure the thread pool size (the default maximum is 200).

The Worker thread in the thread model is responsible for reading the request data from the socket and parsing it to HttpServletRequest , then routing it to servlet (that is, the classic DispatcherServlet ), and finally routing it to the Controller for business calls.

IO read and write cannot be isolated from business operations

• When the business operation is a time-consuming operation, it will occupy Worker thread resources, which will affect the processing of other requests, and also affect the efficiency of IO data reading and writing.
• When network IO read and write operations are time-consuming, it will also affect the efficiency of business execution

There is no good or bad thread model, only suitable and unsuitable

Restful performance loss

Restful style interface design is the most respected interface design by the majority of developers, usually the interface path may be as long as this

• /zoos/{id}
• /zoos/{id}/animals

But the way such an interface is processed in Spring MVC will bring performance loss, because the {id} part is implemented based on regular expressions.

interceptor

When using an interceptor, you can set the matching logic in the following way

• InterceptorRegistration#addPathPatterns("/foo/**", "/fo?/b*r/")
• InterceptorRegistration#excludePathPatterns("/bar/**", "/foo/bar")

Similarly, this feature will also bring a lot of performance consumption of regular expression matching for each request

Only some scenarios are listed here. In fact, there are still many areas in the implementation code of Spring MVC that need to be improved from a performance point of view (of course this is only from a performance point of view...)

Excessive functions of Rest scene

Imagine, when we use Spring Cloud develop micro services, except we use @RequestMapping , @RequestParam common notes and so on, the will to use such ModelAndView , JSP , Freemaker and other related functions it?

Today, when the concept of microservices is familiar, most microservices are no longer an All in One Web service, but multiple Rest-style Web services. This makes Spring MVC, which supports full Servlet JSP and other functions in the All in One scene, seem a little overkill in the Rest scene. Even so, everyone in the Spring Cloud system does not hesitate to use Spring MVC, because Spring Cloud is for us.

Too big

Following the above problem of excess functionality, it will also cause the problem of excessive code and dependency volume. This may not be a big problem in traditional microservice scenarios, but when we mirror it, it will cause the mirror body to be larger. Also in the FaaS scenario, this problem will be magnified, directly affecting the cold start of the function.

FaaS related issues will be discussed later

Lack of standards

The standard here refers to the Rest standard. In fact, there is already a common standard in Java, namely JAX-RS (Java API for RESTful Web Services). JAX-RS was designed for Rest services from the beginning, including some annotations often used in developing Rest services, and A set of Rest services and even client standards.

annotation

JAX-RS

• @Path
• @GET, @POST, @PUT, @DELETE
• @Produces
• @Consumes
• @PathParam
• @QueryParam
• @HeaderParam
• @CookieParam
• @MatrixParam
• @FormParam
• @DefaultValue
• ...

Spring MVC

• @RequestMapping
• @RequestParam
• @RequestHeader
• @PathVariable
• @CookieValue
• @MatrixVariable
• ...

In fact, there is not much difference between JAX-RS annotations and Spring MVC annotations in terms of functionality.

But JAX-RS annotations are compared to Spring MVC annotations

1. more concise : JAX-RS annotation style is more concise, and the form is more unified, while Spring MVC's annotations are all slightly verbose.
2. More flexible: JAX-RS annotations can not only be used on the Controller, @Produces , @Consumes can be used in various places such as serialization and deserialization extension implementations. @DefaultValue annotation can also be used with other annotations. However, @RequestMapping combines various functions in one annotation, and the code is lengthy and complicated.
3. more general : JAX-RS annotations are standard Java annotations that can be used in various environments, @GetMapping @PostMapping and other annotations all rely on Spring's @AliasFor annotations and can only be used in the Spring environment.

For students who are accustomed to Spring MVC, I may not feel it, but the author has personally implemented Spring MVC annotations and JAX-RS compatibility, and I prefer the design of JAX-RS during the whole process.

Tripartite Framework Affinity

If you want to implement an RPC framework and are ready to support RPC calls of the HTTP protocol, imagine that users like Spring Cloud can simply mark some @RequestMapping annotations to complete RPC calls, so now you need a dependency that only contains Spring MVC annotations. Then to implement the corresponding logic. Unfortunately, Spring MVC's annotations are directly coupled to the spring-web dependency. If you want to rely on it, spring-core , spring-beans and other dependencies will be introduced together. Therefore, the HTTP support of the RPC framework in the industry is almost always the choice of JAX-RS (Such as SOFA RPC, Dubbo, etc.).

Not light enough

I have to admit that Spring's code is very design, and the interface design is very elegant.

But a Web service framework such as Spring MVC is a whole, directly attached to the Spring container (perhaps for strategic reasons?). Therefore, all related capabilities need to be introduced into the Spring container, even Spring Boot. Some people may say: "This is not normal. Our projects will all introduce Spring Boot." but

If I am a framework developer, I want to start a web server in my framework to expose the corresponding Http interface, but my framework is very simple and I don’t want to introduce any other dependencies (because it will be passed to the user), this At that time, Spring MVC cannot be used.

If I am a middleware developer, I also want to start a web server in my program to expose the corresponding Metrics interface, but I don’t want to introduce Spring Boot and other related big things because of this function. At this time, I only It can be implemented by itself similar to the native embedded Tomcat or Netty, but this is a bit too complicated (you have to implement it yourself every time).

Introduction to ESA Restlight

Based on the above-mentioned problems and pain points, the ESA Restlight framework was born.

ESA Restlight is a cloud-native high-performance, lightweight web development framework based on Netty.

Hereinafter referred to as Restlight

Quick Start

Create a Spring Boot project and introduce dependencies

<dependency>
    <groupId>io.esastack</groupId>
    <artifactId>restlight-starter</artifactId>
    <version>0.1.1</version>
</dependency>

Write Controller

@RestController
@SpringBootApplication
public class RestlightDemoApplication {

    @GetMapping("/hello")
    public String hello() {
        return "Hello Restlight!";
    }

    public static void main(String[] args) {
        SpringApplication.run(RestlightDemoApplication.class, args);
    }
}

Run the project and visit http://localhost:8080/hello

As you can see, there is almost no difference between using Restlight in Spring Boot and using Spring MVC. The usage is very simple

Performance

testing scenarios

Use Restlight and spring-boot-starter-web (2.3.2.RELEASE) to write two web services respectively, and implement a simple Echo interface (return the body content of the request directly), respectively, in the scenario where the request body is 16B, 128B, 512B, 1KB, 4KB, 10KB carry out testing

test tools

• wrk4.1.0

• 	OS	CPU	Mem(G)
  server	centos:6.9-1.2.5(docker)	4	8
  client	centos:7.6-1.3.0(docker)	16	3

JVM parameters

-server -Xms3072m -Xmx3072m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintTenuringDistribution -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:logs/gc-${appName}-%t.log -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=480M -XX:+UseGCLogFileRotation -XX:HeapDumpPath=.

Parameter configuration

Framework	Options
Restlight	restlight.server.io-threads=8<br/>restlight.server.biz-threads.core=16<br/>restlight.server.biz-threads.max=16<br/>restlight.server.biz-threads.blocking-queue-length=512
Spring Web	server.tomcat.threads.max=32<br/>server.tomcat.accept-count=128

Test result (RPS)

	16B	128B	512B	1KB	4KB	10KB
Restlight(IO)	129457.26	125344.89	125206.74	116963.24	85749.45	49034.57
Restlight(BIZ)	101385.44	98786.62	97622.33	96504.81	68235.2	46460.79
Spring Web	35648.27	38294.94	37940.3	37497.58	32098.65	22074.94

It can be seen that the performance of 2-4 times higher than Spring MVC.

Restlight (IO) and Restlight (BIZ) are unique thread scheduling capabilities in Restlight, using different threading models

Features

• HTTP1.1/HTTP2/H2C/HTTPS support
• SpringMVC and JAX-RS annotation support
• Thread scheduling: arbitrarily schedule Controller to execute in any thread pool
• Enhanced SPI capability: load and filter according to multiple conditions such as grouping, labeling, order, etc.
• Self-protection: CPU overload protection, limit on the number of new connections
• Spring Boot Actuator support
• Fully asynchronous filter, interceptor, exception handler support
• Jackson/Fastjson/Gson/Protobuf serialization support: support serialization negotiation and annotation to specify the serialization method at will
• Compatible with different operating environments: native Java, Spring, and Spring Boot environments can all support
• AccessLog
• IP whitelist
• Fail fast
• Mock test
• ...

ESA Restlight architecture design

Design Principles

• cloud native : quick start, resource saving, lightweight
• High-performance : Continuous pursuit of goals & core competitiveness, based on the high-performance network framework Netty
• High scalability : Open extension points to meet the needs of diversified services
• low access cost : Compatible with SpringMVC and JAX-RS common annotations, reducing user costs
• full link asynchronous: based on CompletableFuture provides perfect asynchronous processing capabilities
• monitoring and statistics: complete thread pool and other indicators monitoring and request link tracking and statistics

Layered architecture design

Through the hierarchical architecture design, Restlight has very high scalability, while providing different implementations for native Java, Spring, Spring Boot and other scenarios, suitable for Spring Boot business, third-party frameworks, middleware, FaaS and other scenarios.

In the architecture diagram, ESA HttpServer , Restlight Server , Restlight Core , Restlight for Spring , Restlight Starter can be used as an independent module to meet the needs of different scenarios

ESA HttpServer

A simple HttpServer based on Netty, supporting Http1.1/Http2 and Https, etc.

The project has been synchronously open sourced to Github: https://github.com/esastack/esa-httpserver

Restlight Server

ESA HttpServer on the basis of 060e40f8c54244

• Introduce business thread pool
• Filter
• Request routing (route the request to the corresponding Handler based on conditions such as url, method, header, etc.)
• Responsive programming support based on CompletableFuture
• Thread scheduling
• …

eg.

Introduce dependencies

<dependency>
    <groupId>io.esastack</groupId>
    <artifactId>restlight-server</artifactId>
    <version>0.1.1</version>
</dependency>

Start an Http Server with one line of code

Restlite.forServer()
        .daemon(false)
        .deployments()
        .addRoute(route(get("/hello"))
                .handle((request, response) ->
                        response.sendResult("Hello Restlight!".getBytes(StandardCharsets.UTF_8))))
        .server()
        .start();

Suitable for various frameworks, middleware and other basic components to start or expect to use code embedded to start HttpServer scenes

Restlight Core

On Restlight Server , the extension supports the Controller method (in the Controller category, the @RequestMappng ) to complete the business logic and many common functions

• HandlerInterceptor : Interceptor
• ExceptionHandler : Global exception handler
• BeanValidation : Parameter verification
• ArgumentResolver : Parameter analysis extension
• ReturnValueResolver : Return value analysis extension
• RequestSerializer : request serializer (usually responsible for deserializing the body content)
• ResposneSerializer : Response serializer (usually responsible for serializing the response object to the Body)
• Built-in Jackson, Fastjson, Gson, ProtoBuf serialization support
• …

Restlight for Spring MVC

Spring MVC annotation support based on Restlight Core

eg

<dependency>
    <groupId>io.esastack</groupId>
    <artifactId>restlight-core</artifactId>
    <version>0.1.1</version>
</dependency>
<dependency>
    <groupId>io.esastack</groupId>
    <artifactId>restlight-jaxrs-provider</artifactId>
    <version>0.1.1</version>
</dependency>

Write Controller

@RequestMapping("/hello")
public class HelloController {

    @GetMapping(value = "/restlight")
    public String restlight() {
        return "Hello Restlight!";
    }
}

Use Restlight to start Server

Restlight.forServer()
        .daemon(false)
        .deployments()
        .addController(HelloController.class)
        .server()
        .start();

Restlight for JAX-RS

JAX-RS annotation support based on Restlight Core

eg.

Introduce dependencies

<dependency>
    <groupId>io.esastack</groupId>
    <artifactId>restlight-core</artifactId>
    <version>0.1.1</version>
</dependency>
<dependency>
    <groupId>io.esastack</groupId>
    <artifactId>restlight-jaxrs-provider</artifactId>
    <version>0.1.1</version>
</dependency>

Write Controller

@Path("/hello")
public class HelloController {

    @Path("/restlight")
    @GET
    @Produces(MediaType.TEXT_PLAIN_VALUE)
    public String restlight() {
        return "Hello Restlight!";
    }
}

Use Restlight start Server

Restlight.forServer()
        .daemon(false)
        .deployments()
        .addController(HelloController.class)
        .server()
        .start();

Restlight for Spring

Based on Restlight Core, it supports automatic configuration of various contents ApplicationContext RestlightOptions , automatic configuration of Filter, Controller, etc. from the container)

Applicable to Spring scenarios

Restlight Starter

Support automatic configuration in Spring Boot scenarios based on Restlight for Spring

Applicable to Spring Boot scenarios

Restlight Actuator

Based on Restlight Starter, it supports various Endpoints natively in Spring Boot Actuator and Endpoints unique to Restlight.

Applicable to Spring Boot Actuator scenarios

Thread model

Restlight uses Netty as the implementation of the underlying HttpServer, so part EventLoop is used in the figure. The thread model consists of Acceptor , IO EventLoopGroup (IO thread pool) and Biz ThreadPool (business thread pool).

• Acceptor : A thread pool composed of 1 thread, responsible for monitoring the local port and distributing IO events.
• IO EventLoopGroup : Composed of multiple threads, responsible for reading and writing IO data (corresponding to read() and write() ), as well as the encoding and decoding of the HTTP protocol and the work of distributing it to the service thread pool.
• Biz Scheduler : Responsible for executing the real business logic (mostly business processing in the Controller, interceptors, etc.).
• Custom Scheduler : custom thread pool

Through the addition of the third thread pool Biz Scheduler to complete the asynchronous between IO operations and actual business operations (at the same time, it can be freely scheduled through the thread scheduling function of Restlight)

Flexible thread scheduling & interface isolation

Thread scheduling allows users to freely specify whether the Controller executes on the IO thread, executes on the Biz thread, or executes on a custom thread as needed.

Specify to run on the IO thread

@RequestMapping("/hello")
@Scheduled(Schedulers.IO)
public String list() {
    return "Hello";
}

Specify to execute in the BIZ thread pool

@RequestMapping("/hello")
@Scheduled(Schedulers.BIZ)
public String list() {
    // ...
    return "Hello";
}

Specify to execute in a custom thread pool

@RequestMapping("/hello")
@Scheduled("foo")
public String list() {
    // ...
    return "Hello";
}

@Bean
public Scheduler scheduler() {
    // 注入自定义线程池
    return Schedulers.fromExecutor("foo", Executors.newCachedThreadPool());
}

Through random thread scheduling, users can balance thread switching and isolation to achieve optimal performance or isolation effects

Some details of ESA Restlight performance optimization

Restlight always puts performance in the first place, even sometimes to the point of being paranoid about performance.

Netty

Restlight is written based on Netty. Some of the high-performance features of Netty are naturally the cornerstones of high-performance. The common features of Netty are all used in Restlight.

• Epoll & NIO
• ByteBuf
• PooledByteBufAllocator
• EventLoopGroup
• Future & Promise
• FastThreadLocal
• InternalThreadLocalMap
• Recycler
• ...

In addition to doing a lot of other work

HTTP protocol codec optimization

When it comes to realize Http protocol codec in Netty, the most common usage is HttpServerCodec + HttpObjectAggregator combination (or the HttpRequestDecoder + HttpResponseEncoder + HttpObjectAggregator combination).

Take Http1.1 as an example

In fact, HttpServerCodec has completed the encoding and decoding of the Http protocol, but HttpObjectAggregator is the role of 060e40f8c54ce2?

HttpServerCodec will parse the Http protocol as HttpMessage (the request is HttpRequest , and the response is HttpResponse ), HttpContent , LastHttpContent three parts representing the status line and the header of the HTTP protocol (including the request line) The last body data block (used to identify the request/response end, and also contains Trailer data).

Take request analysis as an example. Usually, what we need is a complete request, not a single HttpRequest , or a body message body HttpContent . Therefore, HttpObjectAggregator is to HttpServerCodec HttpRequest , HttpContent and LastHttpContent that 060e40f8c54d4e parsed into one FullHttpRequest , which is convenient for users to use.

But HttpObjectAggregator still has some problems

• maxContentLength question

  HttpObjectAggregator constructor need to specify a maxContentLength parameter that specifies when the body is thrown aggregation request is too large TooLongFrameException . The problem is that this parameter is of int , so the size of the requested Body cannot exceed the maximum value of int 2^31-1, which is 2G. In scenes such as large files, large bodies, chunks, etc., it is counterproductive.

• performance

  Usually although we need an integrated FullHttpRequest parsing result, in fact, when we pass the request object backwards, we can't directly give the Netty native object to the user, so most of them need to do a package by HttpServletRequest (for example, similar to 060e40f8c54e59). The original HttpServerCodec parsed result is converted twice, the first time is converted to FullHttpRequest , and the second time is converted to a user-defined object. In fact, what we really need is to wait for the decoding of the entire Http protocol to be completed and aggregate the results into our own objects.

• Big body problem

  Aggregation means that you have to wait until all the bodies have been received before you can do subsequent operations. However, if it is a Multipart request and the request contains large files, using HttpObjectAggregator this time will keep all the body data in the memory ( Or even direct memory) until the end of this request. This is almost unacceptable.

  Generally, there are two solutions to this scenario: 1) Dump the received body data to the local disk, release memory resources, and read the disk data through the stream when it is needed. 2) Every time a part of body data is received, it is consumed and released immediately.

  Both of these methods require that the requested body cannot be directly aggregated.

• Responsive body processing

  For the Http protocol, although these are usually steps:

  Client sends complete request -> server receives complete request -> server sends complete response -> client receives complete response

  But in fact, we can be more flexible. When processing a request, every time a body is received, it will be directly handed over to the business.

  Client sends complete request -> server receives request header -> server processes body1 -> server processes body2 -> server processes body3 -> server sends complete response

  We even managed to send and process the body in response to the client and server at the same time

Therefore, we implemented the aggregation logic Http1Handler and Http2Handler

• Responsive body processing

  HttpServer.create()
          .handle(req -> {
              req.onData(buf -> {
                  // 每收到一部分的body数据都将调用此逻辑
                  System.out.println(buf.toString(StandardCharsets.UTF_8));
              });
              req.onEnd(p -> {
                  // 写响应
                  req.response()
                          .setStatus(200)
                          .end("Hello ESA Http Server!".getBytes(StandardCharsets.UTF_8));
                  return p.setSuccess(null);
              });
          })
          .listen(8080)
          .awaitUninterruptibly();

• Get the entire request

  HttpServer.create()
          .handle(req -> {
              // 设置期望聚合所有的body体
              req.aggregate(true);
              req.onEnd(p -> {
                  // 获取聚合后的body
                  System.out.println(req.aggregated().body().toString(StandardCharsets.UTF_8));
                  // 写响应
                  req.response()
                          .setStatus(200)
                          .end("Hello ESA Http Server!".getBytes());
                  return p.setSuccess(null);
              });
          })
          .listen(8080)
          .awaitUninterruptibly();

• Responsive request body processing and response body processing

  HttpServer.create()
          .handle(req -> {
              req.onData(buf -> {
                  // 每收到一部分的body数据都将调用此逻辑
                  System.out.println(buf.toString(StandardCharsets.UTF_8));
              });
              req.onEnd(p -> {
                  req.response().setStatus(200);
                  // 写第一段响应body
                  req.response().write("Hello".getBytes(StandardCharsets.UTF_8));
                  // 写第二段响应body
                  req.response().write(" ESA Http Server!".getBytes(StandardCharsets.UTF_8));
                  // 结束请求
                  req.response().end();
                  return p.setSuccess(null);
              });
          })
          .listen(8080)
          .awaitUninterruptibly();

performance

test scene

Use ESA HttpServer and native Netty ( HttpServerCodec , HttpObjectAggregator ) to write two web services to implement a simple Echo interface (return the body content of the request directly), respectively, in the scenario where the request body is 16B, 128B, 512B, 1KB, 4KB, 10KB carry out testing

test tools

• wrk4.1.0

• 	OS	CPU	Mem(G)
  server	centos:6.9-1.2.5(docker)	4	8
  client	centos:7.6-1.3.0(docker)	16	3

JVM parameters

-server -Xms3072m -Xmx3072m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=256m -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintTenuringDistribution -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:logs/gc-${appName}-%t.log -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=480M -XX:+UseGCLogFileRotation -XX:HeapDumpPath=.

Parameter configuration

The number of IO threads is set to 8

Test result (RPS)

	16B	128B	512B	1KB	4KB	10KB
Netty	133272.34	132818.53	132390.78	127366.28	85408.7	49798.84
ESA HttpServer	142063.99	139608.23	139646.04	140159.5	92767.53	53534.21

The performance of using ESA HttpServer is even higher than that of native Netty

Route cache

In traditional Spring MVC, when our @RequestMapping annotation contains any complex matching logic (the complex logic here can be understood as except for a url corresponding to a Controller implementation, and there is no pattern matching such as *,?. {Foo} in the url Content) time can have a relatively good effect in the routing stage. On the contrary, as in normal circumstances, a request arrives and is routed to the corresponding Controller. This process will be traversed and matched in all Note that this type of traversal is almost unavoidable in the general environment where microservices advocate RestFul design. At the same time, due to the complexity of the matching condition itself (for example, the regularity itself is criticized for performance), it is accompanied by The routing loss of SpringMVC is very large.

Cache design

• Twenty-eight principles (80% of the business is handled by 20% of the interface)
• Algorithm: LFU (Least Frequently Used) algorithm

Although we cannot change the loss of the routing condition matching itself, we hope to do as few matching times as possible to achieve the optimization effect. Therefore, the commonly used "cache" is used as an optimization method.
When routing caching is enabled, by default, the LFU (Least Frequently Used) algorithm will be used to cache 10% of the Controllers. According to the 28th principle (80% of the business is handled by 20% of the interface), most of the requests are Will match successfully in the cache and return (here the default cache of the framework is one-tenth, which is a relatively conservative setting)

Algorithm logic

When each request is matched successfully, the operation of adding 1 to the hit record will be performed, and the 20% (configurable) Controller with the highest hit record will be added to the cache. Each time a request comes, the matching Controller will be searched from the cache ( Most requests will be returned at this stage), if it fails, it will enter the logic of normal matching.

When will the cache be updated? We will not update the cache every time a request hits, because this involves a sorting (or m traversals, where m is the number of Controllers that need to be cached, which is equivalent to picking out the m controllers with the highest hits). Instead, we will recalculate and update the cache in a probabilistic manner. According to the 2-8 principle, usually our current cache memory is what we need, so there is no need to recalculate and update the cache every time a request hits. Therefore, we will take this operation under the condition of a certain probability of the request hit (default 0.1%, called the calculated probability), reducing the concurrency loss (this logic itself is based on CopyOnWrite, and is purely lock-free concurrent programming, itself Performance loss is very low). At the same time, this probability is configurable, and the configuration can be adjusted according to the actual situation of the specific application to achieve the optimal effect.

effect

Use JMH for micro-benchmark test, and compare the performance test between the operation with cache and without cache

Test the performance when the number of Controllers is 10, 20, 50, and 100 respectively

Request to obey Poisson distribution , 5 rounds of warm-up, 10 iterations per test

@BenchmarkMode({Mode.Throughput})
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Warmup(iterations = 5)
@Measurement(iterations = 10)
@Threads(Threads.MAX)
@Fork(1)
@State(Scope.Benchmark)
public class CachedRouteRegistryBenchmark {

    private ReadOnlyRouteRegistry cache;
    private ReadOnlyRouteRegistry noCache;

    @Param({"10", "20", "50", "100"})
    private int routes = 100;

    private AsyncRequest[] requests;
    private double lambda;

    @Setup
    public void setUp() {
        RouteRegistry cache = new CachedRouteRegistry(1);
        RouteRegistry noCache = new SimpleRouteRegistry();
        Mapping[] mappings = new Mapping[routes];
        for (int i = 0; i < routes; i++) {
            HttpMethod method = HttpMethod.values()[ThreadLocalRandom.current().nextInt(HttpMethod.values().length)];
            final MappingImpl mapping = Mapping.mapping("/f?o/b*r/**/??x" + i)
                    .method(method)
                    .hasParam("a" + i)
                    .hasParam("b" + i, "1")
                    .hasHeader("c" + i)
                    .hasHeader("d" + i, "1")
                    .consumes(MediaType.APPLICATION_JSON)
                    .produces(MediaType.TEXT_PLAIN);
            mappings[i] = mapping;
        }

        for (Mapping m : mappings) {
            Route route = Route.route(m);
            cache.registerRoute(route);
            noCache.registerRoute(route);
        }

        requests = new AsyncRequest[routes];
        for (int i = 0; i < requests.length; i++) {
            requests[i] = MockAsyncRequest.aMockRequest()
                    .withMethod(mappings[i].method()[0].name())
                    .withUri("/foo/bar/baz/qux" + i)
                    .withParameter("a" + i, "a")
                    .withParameter("b" + i, "1")
                    .withHeader("c" + i, "c")
                    .withHeader("d" + i, "1")
                    .withHeader(HttpHeaderNames.CONTENT_TYPE.toString(), MediaType.APPLICATION_JSON.value())
                    .withHeader(HttpHeaderNames.ACCEPT.toString(), MediaType.TEXT_PLAIN.value())
                    .build();
        }
        this.cache = cache.toReadOnly();
        this.noCache = noCache.toReadOnly();
        this.lambda = (double) routes / 2;
    }

    @Benchmark
    public Route matchByCachedRouteRegistry() {
        return cache.route(getRequest());
    }

    @Benchmark
    public Route matchByDefaultRouteRegistry() {
        return noCache.route(getRequest());
    }

    private AsyncRequest getRequest() {
        return requests[getPossionVariable(lambda, routes - 1)];
    }

    private static int getPossionVariable(double lambda, int max) {
        int x = 0;
        double y = Math.random(), cdf = getPossionProbability(x, lambda);
        while (cdf < y) {
            x++;
            cdf += getPossionProbability(x, lambda);
        }
        return Math.min(x, max);
    }

    private static double getPossionProbability(int k, double lamda) {
        double c = Math.exp(-lamda), sum = 1;
        for (int i = 1; i <= k; i++) {
            sum *= lamda / i;
        }
        return sum * c;
    }
}

Test Results

Benchmark                                                 (routes)   Mode  Cnt     Score    Error   Units
CachedRouteRegistryBenchmark.matchByCachedRouteRegistry         10  thrpt   10  1353.846 ± 26.633  ops/ms
CachedRouteRegistryBenchmark.matchByCachedRouteRegistry         20  thrpt   10   982.295 ± 26.771  ops/ms
CachedRouteRegistryBenchmark.matchByCachedRouteRegistry         50  thrpt   10   639.418 ± 22.458  ops/ms
CachedRouteRegistryBenchmark.matchByCachedRouteRegistry        100  thrpt   10   411.046 ±  5.647  ops/ms
CachedRouteRegistryBenchmark.matchByDefaultRouteRegistry        10  thrpt   10   941.917 ± 33.079  ops/ms
CachedRouteRegistryBenchmark.matchByDefaultRouteRegistry        20  thrpt   10   524.540 ± 18.628  ops/ms
CachedRouteRegistryBenchmark.matchByDefaultRouteRegistry        50  thrpt   10   224.370 ±  9.683  ops/ms
CachedRouteRegistryBenchmark.matchByDefaultRouteRegistry       100  thrpt   10   113.883 ±  5.847  ops/ms

It can be seen that the performance has improved significantly after adding the cache. At the same time, it can be seen that as the number of Controllers increases, the performance loss of the scene without cache is very serious.

Interceptor design

Spring MVC interceptor performance issues

As mentioned earlier, the interceptor in Spring MVC will cause performance problems due to regular expression problems. Restlight has introduced different types of interceptors while optimizing the performance of regular matching.

Imagine whether there will be the following scenarios in SpringMVC

scene 1

If you want to intercept a Controller, its Path is /foo addPathPatterns("/foo") will be used to intercept

Such a scenario is relatively simple, Spring MVC only needs to perform direct Uri matching, and the performance consumption is not large.

scene 2

If you want to intercept all Controllers in a Controller Class, they have a common prefix. At this time, you may use addPathPatterns("/foo/**") intercept

At this time, it is necessary to perform a regular matching on all requests, and the performance loss is large

Scene 3

If you want to intercept multiple Controllers with different prefixes and exclude a few of them at the same time, you may need to use addPathPatterns("/foo/**", "/bar/***") and excludePathPatterns("/foo/b*", "/bar/q?x")

At this time, it is necessary to perform multiple regular matching on all requests, and the performance loss is different according to the regular complexity, and the impact is relatively large

Interceptor design in Restlight

The fundamental purpose of interceptor design is to allow users to intercept the target Controller as they like.

RouteInterceptor

Only bound to the fixed Controller/Route interceptor. This interceptor allows the user to decide which Controller to intercept initialization phase does not perform any and is directly bound to this Controller.

At the same time, the Controller metadata information is directly used as a parameter, and the user does not need to be limited to URL path matching. The user can match according to various information such as annotations, HttpMethod, Uri, method signatures, etc.

In Restlight, a Controller interface is abstracted as a Route

eg.

Implement an interceptor to intercept all GET requests (only GET)

@Bean
public RouteInterceptor interceptor() {
    return new RouteInterceptor() {

        @Override
        public CompletableFuture<Boolean> preHandle0(AsyncRequest request,
                                                     AsyncResponse response,
                                                     Object handler) {
            // biz logic
            return CompletableFuture.completedFuture(null);
        }

        @Override
        public boolean match(DeployContext<? extends RestlightOptions> ctx, Route route) {
            HttpMethod[] method = route.mapping().method();
            return method.length == 1 && method[0] == HttpMethod.GET;
        }
    };
}

MappingInterceptor

Bind all Controller/Route and match the requested interceptor.

Users can match arbitrarily according to the request, not limited to Uri, and the performance is higher.

eg.

Implement an interceptor to intercept all X-Foo request header in the Header

@Bean
public MappingInterceptor interceptor() {
    return new MappingInterceptor() {

        @Override
        public CompletableFuture<Boolean> preHandle0(AsyncRequest request,
                                                     AsyncResponse response,
                                                     Object handler) {
            // biz logic
            return CompletableFuture.completedFuture(null);
        }
        
        @Override
        public boolean test(AsyncRequest request) {
            return request.containsHeader("X-Foo");
        }
    };
}

Canonical intersection optimization

The above interceptor design is to solve the performance problem of regular expressions from the design stage, but if the user just wants to use the same way as the Spring MVC interceptor.

Therefore, we need to face the performance problem of interceptor Uri matching.

HandlerInterceptor

Interceptor compatible with Spring MVC usage

• includes() : Path that specifies the scope of the interceptor, which acts on all requests by default.
• excludes() : Path specified by the interceptor (priority higher than includes ) is empty by default.

eg.

Implement an interceptor to intercept all /foo/ beginning with /foo/bar

@Bean
public HandlerInterceptor interceptor() {
    return new HandlerInterceptor() {

        @Override
        public CompletableFuture<Boolean> preHandle0(AsyncRequest request,
                                                     AsyncResponse response,
                                                     Object handler) {
            // biz logic
            return CompletableFuture.completedFuture(null);
        }

        @Override
        public String[] includes() {
            return new String[] {"/foo/**"};
        }

        @Override
        public String[] excludes() {
            return new String[] {"/foo/bar"};
        }
    };
}

This kind of interceptor is actually not much different from Spring MVC in terms of function, and it is matched by Uri.

regular intersection judgment

Imagine, now I wrote a Controller /foo/bar

• includes("/foo/**")

For the Controller, in fact, the interceptor will be 100% match to the interceptor, because /foo/** this regular is included /foo/bar of

same

• includes("/foo/b?r")
• includes("/foo/b*")
• includes("/f？o/b*r")

This series of matching rules will definitely match

on the contrary

• excludes("/foo/**")

It will definitely not match

optimized logic

• When the interceptor's includes() and excludes() rules must to the Controller, it is directly bound to the Controller during the initialization phase, and no matching operation is performed during runtime.
• When the interceptor's includes() and excludes() rules must not to the Controller, it will be ignored in the initialization phase, and no matching operation will be performed during runtime.
• Interceptor includes() and excludes() may when will match the Controller, matching runtime

We judge the intersection between the interceptor rules and the Controller in the program startup phase, and put the matching logic in the startup phase to complete it at one time, which greatly improves the performance of each request.

In fact, when may match the Controller, Restlight will further optimize it, so the space is limited and I won’t go into details...

Restful design no longer worry about performance

As mentioned earlier, the use /zoos/{id} in Spring MVC will cause performance loss due to regularization. This problem will no longer exist in Restlight.

Reference PR

Restlight as FaaS Runtime

Faas

FaaS (Functions as a Service), which is a hot vocabulary of cloud native now, belongs to the category of Serverless.

The core of Serverless

• Pay for what you use
• On-demand
• Fast elastic scaling
• Event driven
• Non-local state persistence
• Resource maintenance custody

Among them, is a thorny issue for FaaS scenarios.

The most prominent problem is the cold start problem. After the Pod is reduced to 0, when a new request comes in, it is necessary to schedule a new Pod to provide services as soon as possible.

This problem is particularly prominent in Knative. Due to the use of KPA for scheduling expansion and contraction, the cold start time is longer, so I will not discuss it here.

Fission

Fission is another solution for FaaS. FaaS scenarios are very sensitive to the cold start time, while Fission uses hot Pod pool technology to solve the cold start problem.

By pre-starting a set of hot Pod pools, pre-starting the resources below the user logic such as mirrors, JVMs, and Web containers, and hot loading Function codes and providing services during expansion, the cold start time is shortened to less than 100ms (Knative may require 10s or even 30s time).

Just take Fission as an example, the Fission solution is not mature yet, we make in-depth modifications and enhancements to it internally

The challenge of the frame

One of the most common scenarios in FaaS is HttpTrigger, that is, users write an Http interface (or Controller), and then rely on this piece of code to run in a web container.

With the hot Pod pool technology, the cold start time is more in the process of specialization (loading the Function code, exposing the Http service in the already running Pod).

Cold start

• The startup speed itself is fast enough
• The application size is small enough (save the time of mirroring pull)
• Less resource usage (less CPU, memory usage)

standard

Users don’t need to pay attention when writing functions, nor should they pay attention to whether the actual FaaS underlying Http service uses Spring MVC or Restlight or other components. Therefore, users should not be required to use Spring MVC to write Http interfaces. At this time, they need to be defined. A set of standards that shield the details of the underlying basic settings, allowing users to write functions without any other dependencies.

JAX-RS is a better choice (of course it is not the only choice)

monitoring indicators

FaaS requires rapid expansion and contraction. The most direct basis for judging whether a service needs expansion or contraction is Metrics. Therefore, more clear indicators need to be exposed within the framework to allow FaaS to perform rapid expansion and contraction responses. For example: thread pool usage, queuing, thread pool rejection and other indicators.

Restlight

Obviously Spring MVC cannot meet this scenario, because it is designed for long-running services and relies on many components such as Spring Boot. It has a large body and startup speed that cannot meet the requirements of cold start.

Restlight fits the FaaS scenario very well.

• Start fast
• Small size: does not rely on any tripartite dependence
• Rich indicators: IO threads, Biz thread pool indicators
• No environment dependency: pure native Java can be started
• Support JAX-RS
• High performance: A single Pod can carry more concurrent requests, saving costs

Now our company has used Restlight as the FaaS Java Runtime base to build FaaS capabilities.

Restlight future planning

JAX-RS full support

At this stage, Restlight only supports JAX-RS annotations, and will support the entire JAX-RS specification in the future.

This is very meaningful, JAX-RS is a standard designed specifically for Rest services, which is consistent with the starting point of Restlight at the beginning.

At the same time last year, JAX-RS has released JAX-RS 3.0 , and now there are few frameworks in the industry that support it, and Restlight will take the lead to support it.

FaaS Runtime in-depth support

As the base of FaaS Runtimme, Restlight needs more and lower-level capabilities.

Function is currently in exclusive Pod mode. For low-frequency access functions, the Pod instance is reserved for waste, and reduced to 0 will be frequently cold-started. At present, the only option is to reduce the Pod specifications as much as possible and increase the idle time of the Pod.

Ideally, we hope that Pod can support the operation of multiple functions at the same time, which can save more costs. But this has higher requirements for Function isolation

So Restlight will support in the future

• Dynamic Route: Dynamically modify the Route in the web container at runtime to meet the requirements of runtime specialization
• Coroutine support: Run Function in a lighter way, reducing resource contention
• Route isolation: Meet the isolation requirements between different functions, and prevent one function from affecting other functions
• Resource billing: how many resources are used by different functions
• More refined Metrics: more accurate and timely indicators to meet the needs of rapid expansion and contraction.

Native Image support

Cloud native also puts forward more requirements on traditional microservices, requiring services to be small in size and quick to start.

Therefore, Restlight will also consider supporting Native Image and directly compiling it into a binary file, thereby improving startup speed and reducing resource consumption.

After the actual measurement of Graal VM, the effect is not so ideal, and it is not very friendly to use.

Concluding remarks

Restlight focuses on cloud-native Rest service development.

Unwavering in the cloud-native direction

The ultimate pursuit of performance

Have a cleanliness of code

It is also a young project, and all technical enthusiasts are welcome to join in to discuss learning and progress together.

Author profile

Norman OPPO Senior Backend Engineer

Focus on cloud-native microservices, cloud-native framework, ServiceMesh, Serverless and other technologies.

For more exciting content, welcome to pay attention to the [OPPO Internet Technology] public account