I am a request, how was I sent?

Abstract: This article mainly analyzes the scenario of using the RestTemplate provided by cse. In fact, the final call logic of the rpc annotation (RpcReference) provided by cse is the same as the RestTemplate.

This article is shared from the HUAWEI CLOUD community " I am a request, where should I go (below) ", the original author: Xiang Hao.

Last time we probably learned how the server handles the request, so what is the process of sending the request? This article mainly analyzes the scenario of using the RestTemplate provided by cse. In fact, the final call logic of the rpc annotation (RpcReference) provided by cse is the same as that of RestTemplate.

use

When using the RestTemplate provided by cse, it is initialized like this:

RestTemplate restTemplate = RestTemplateBuilder.create();

restTemplate.getForObject("cse://appId:serviceName/xxx", Object.class);

We can notice 2 weird places:

• RestTemplate is obtained through RestTemplateBuilder.create() instead of the one provided in Spring.
• The request path starts with cse instead of our common http and https and needs to add the application ID and service name to which the service belongs.

Parsing

Match RestTemplate according to url

First look at RestTemplateBuilder.create(), it returns org.apache.servicecomb.provider.springmvc.reference.RestTemplateWrapper, which is a wrapper class provided by cse.

// org.apache.servicecomb.provider.springmvc.reference.RestTemplateWrapper
// 用于同时支持cse调用和非cse调用
class RestTemplateWrapper extends RestTemplate {
    private final List<AcceptableRestTemplate> acceptableRestTemplates = new ArrayList<>();
 
    final RestTemplate defaultRestTemplate = new RestTemplate();
 
    RestTemplateWrapper() {
        acceptableRestTemplates.add(new CseRestTemplate());
    }
 
    RestTemplate getRestTemplate(String url) {
        for (AcceptableRestTemplate template : acceptableRestTemplates) {
            if (template.isAcceptable(url)) {
                return template;
            }
        }
        return defaultRestTemplate;
    }
}

AcceptableRestTemplate: This class is an abstract class that also inherits RestTemplate. At present, its subclass is CseRestTemplate. We can also see that a CseRestTemplate will be added to AcceptableRestTemplates by default during initialization.

Back to the place of use restTemplate.getForObject: This method will delegate to the following method:

public <T> T getForObject(String url, Class<T> responseType, Object... urlVariables) throws RestClientException {
    return getRestTemplate(url).getForObject(url, responseType, urlVariables);
}

It can be seen that getRestTemplate(url) will be called first, that is, template.isAcceptable(url) will be called. If it matches, it will return CseRestTemplate, otherwise it will return to the regular RestTemplate. Then look at the isAcceptable() method:

Here we understand the role of cse:// in the path, just to use CseRestTemplate to initiate requests, and understand why RestTemplateWrapper can support both cse calls and non-cse calls.

Delegate call

From the above, we can see that our cse calls are actually delegated to CseRestTemplate. Several things are initialized when constructing CseRestTemplate:

public CseRestTemplate() {
    setMessageConverters(Arrays.asList(new CseHttpMessageConverter()));
    setRequestFactory(new CseClientHttpRequestFactory());
    setUriTemplateHandler(new CseUriTemplateHandler());
}

Here you need to focus on new CseClientHttpRequestFactory():

public class CseClientHttpRequestFactory implements ClientHttpRequestFactory {
    @Override
    public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
        return new CseClientHttpRequest(uri, httpMethod);
    }
}

finally delegated to this class CseClientHttpRequest, here is the highlight!

Let's first turn our attention back to this sentence: restTemplate.getForObject("cse://appId:serviceName/xxx", Object.class), from the above we know that the logic is to find the corresponding RestTemplate first according to the url, and then call The getForObject method will eventually be called to: org.springframework.web.client.RestTemplate#doExecute:

protected <T> T doExecute(URI url, @Nullable HttpMethod method, @Nullable RequestCallback requestCallback,
    @Nullable ResponseExtractor<T> responseExtractor) throws RestClientException {
 
    ClientHttpResponse response = null;
    try {
        ClientHttpRequest request = createRequest(url, method);
        if (requestCallback != null) {
            requestCallback.doWithRequest(request);
        }
        response = request.execute();
        handleResponse(url, method, response);
        return (responseExtractor != null ? responseExtractor.extractData(response) : null);
    }
}

createRequest(url, method): getRequestFactory().createRequest(url, method) will be called, that is, it will eventually be called to the RequestFactory that we initialized CseClientHttpRequest, so the ClientHttpRequest class will be returned here.

request.execute(): This method will delegate to the org.apache.servicecomb.provider.springmvc.reference.CseClientHttpRequest#execute method.

So far we know that the previous call will eventually be delegated to the CseClientHttpRequest#execute method.

cse call

Then the above analysis:

public ClientHttpResponse execute() {
    path = findUriPath(uri);
    requestMeta = createRequestMeta(method.name(), uri);
 
    QueryStringDecoder queryStringDecoder = new QueryStringDecoder(uri.getRawSchemeSpecificPart());
    queryParams = queryStringDecoder.parameters();
 
    Object[] args = this.collectArguments();
 
    // 异常流程，直接抛异常出去
    return this.invoke(args);
}

createRequestMeta(method.name(), uri): This is mainly to obtain the information of calling the service according to the microserviceName, and put the obtained information into the Map. The service information is as follows:

You can see that the information inside is very rich, such as application name, service name, and yaml information corresponding to the interface.

this.collectArguments(): There is a checkpoint hidden here, which is to check whether the incoming parameters conform to the definition of the other party's interface. Mainly through this method: org.apache.servicecomb.common.rest.codec.RestCodec#restToArgs, if it does not meet the real process, it ends.

Prepare for invocation

It can be seen from the above analysis that this method will be called after obtaining the parameters required by the interface: org.apache.servicecomb.provider.springmvc.reference.CseClientHttpRequest#invoke:

private CseClientHttpResponse invoke(Object[] args) {
    Invocation invocation = prepareInvocation(args);
    Response response = doInvoke(invocation);
 
    if (response.isSuccessed()) {
        return new CseClientHttpResponse(response);
    }
 
    throw ExceptionFactory.convertConsumerException(response.getResult());
}

prepareInvocation(args): This method will prepare the Invocation. This Invocation has been analyzed in the previous episode, but it is for the server in the previous episode, so of course we have to work for the consumer.

protected Invocation prepareInvocation(Object[] args) {
    Invocation invocation =
        InvocationFactory.forConsumer(requestMeta.getReferenceConfig(),
            requestMeta.getOperationMeta(),
            args);
 
    return invocation;
}

It can also be seen from the name that it serves the consumer side. In fact, whether it is forProvider or forConsumer, the main difference between them is that the loaded Handler is different. The loaded Handler this time is as follows:

• class org.apache.servicecomb.qps.ConsumerQpsFlowControlHandler (flow control)
• class org.apache.servicecomb.loadbalance.LoadbalanceHandler (load)
• class org.apache.servicecomb.bizkeeper.ConsumerBizkeeperHandler (fault tolerance)
• class org.apache.servicecomb.core.handler.impl.TransportClientHandler (call, loaded by default)

The first 3 Handlers can refer to this governance column

doInvoke(invocation): After the invocation is initialized, it starts to be called. This method will eventually be called: org.apache.servicecomb.core.provider.consumer.InvokerUtils#innerSyncInvoke

So far, these actions are the difference between RestTemplate and rpc calls in cse. However, it can be clearly seen that the RestTemplate method only supports synchronization, that is, innerSyncInvoke, but rpc can support asynchronous, that is, reactiveInvoke

public static Response innerSyncInvoke(Invocation invocation) {
 
    invocation.next(respExecutor::setResponse);
}

At this point, we know that the consumer's request is still driven by the invocation chain of responsibility.

Start the invocation responsibility chain

Well, our old friend has appeared again: invocation.next, this method is a typical responsibility chain model, and the chain is the 4 Handlers mentioned above. The first 3 will not be analyzed, and jump directly to TransportClientHandler.

// org.apache.servicecomb.core.handler.impl.TransportClientHandler
public void handle(Invocation invocation, AsyncResponse asyncResp) throws Exception {
    Transport transport = invocation.getTransport();
    transport.send(invocation, asyncResp);
}

invocation.getTransport(): Get the request address, that is, when the request is finally sent, it is still in the form of ip:port.

transport.send(invocation, asyncResp): The call chain is

org.apache.servicecomb.transport.rest.vertx.VertxRestTransport#send

• ->org.apache.servicecomb.transport.rest.client.RestTransportClient#send (HttpClientWithContext will be initialized here and will be analyzed below)
• ->org.apache.servicecomb.transport.rest.client.http.RestClientInvocation#invoke (where the request is actually sent)

public void invoke(Invocation invocation, AsyncResponse asyncResp) throws Exception {
 
    createRequest(ipPort, path);
    clientRequest.putHeader(org.apache.servicecomb.core.Const.TARGET_MICROSERVICE, invocation.getMicroserviceName());
    RestClientRequestImpl restClientRequest =
        new RestClientRequestImpl(clientRequest, httpClientWithContext.context(), asyncResp, throwableHandler);
    invocation.getHandlerContext().put(RestConst.INVOCATION_HANDLER_REQUESTCLIENT, restClientRequest);
 
    Buffer requestBodyBuffer = restClientRequest.getBodyBuffer();
    HttpServletRequestEx requestEx = new VertxClientRequestToHttpServletRequest(clientRequest, requestBodyBuffer);
    invocation.getInvocationStageTrace().startClientFiltersRequest();
    // 触发filter.beforeSendRequest方法
    for (HttpClientFilter filter : httpClientFilters) {
        if (filter.enabled()) {
            filter.beforeSendRequest(invocation, requestEx);
        }
    }
 
    // 从业务线程转移到网络线程中去发送
    // httpClientWithContext.runOnContext
}

createRequest(ipPort, path): Initialize the HttpClientRequest clientRequest according to the parameters. When initializing, a responseHandler will be passed in to create a responseHandler, which is the processing of the response.

Note that the call of org.apache.servicecomb.common.rest.filter.HttpClientFilter#afterReceiveResponse is here foreshadowing, which is triggered by the callback org.apache.servicecomb.transport.rest.client.http.RestClientInvocation#processResponseBody method (Created when creating responseHandler)

And org.apache.servicecomb.common.rest.filter.HttpClientFilter#beforeSendRequest: The trigger of this method can also clearly see the execution of the sending request.

requestEx: Note that its type is HttpServletRequestEx. Although there is a Servlet in the name, you can find that many of the commonly used methods in tomcat throw exceptions directly. This is also a point of error!

httpClientWithContext.runOnContext: The logic used to send the request, but it is still a bit circumscribed here, the following is a key analysis

httpClientWithContext.runOnContext

First look at the definition of HttpClientWithContext:

public class HttpClientWithContext {
    public interface RunHandler {
        void run(HttpClient httpClient);
    }
 
    private HttpClient httpClient;
 
    private Context context;
 
    public HttpClientWithContext(HttpClient httpClient, Context context) {
        this.httpClient = httpClient;
        this.context = context;
    }
 
    public void runOnContext(RunHandler handler) {
        context.runOnContext((v) -> {
            handler.run(httpClient);
        });
    }
}

It can be seen from the above that this method is called to send the request: runOnContext, the parameter is the RunHandler interface, and then it is passed in in the form of lambda, the parameter of lambda is httpClient, and this httpClient is initialized in the constructor of HttpClientWithContext. This constructor is initialized in the method org.apache.servicecomb.transport.rest.client.RestTransportClient#send (call org.apache.servicecomb.transport.rest.client.RestTransportClient#findHttpClientPool method).

But we observe the calling place:

// 从业务线程转移到网络线程中去发送
httpClientWithContext.runOnContext(httpClient -> {
    clientRequest.setTimeout(operationMeta.getConfig().getMsRequestTimeout());
    processServiceCombHeaders(invocation, operationMeta);
    try {
        restClientRequest.end();
    } catch (Throwable e) {
        LOGGER.error(invocation.getMarker(),
            "send http request failed, local:{}, remote: {}.", getLocalAddress(), ipPort, e);
        fail((ConnectionBase) clientRequest.connection(), e);
    }
});

In fact, HttpClient is not used in this piece of logic. In fact, the action of sending a request is triggered by restClientRequest.end(), restClientRequest is the class RestClientRequestImpl in cse, and then it wraps HttpClientRequest (provided in vertx), namely restClientRequest.end() finally delegated to HttpClientRequest.end().

So how is this HttpClientRequest initialized? It is initialized in the method createRequest(ipPort, path), that is, at the entrance of calling the org.apache.servicecomb.transport.rest.client.http.RestClientInvocation#invoke method.

The initialization logic is as follows:

clientRequest = httpClientWithContext.getHttpClient().request(method, 
requestOptions, this::handleResponse)

httpClientWithContext.getHttpClient(): This method returns HttpClient. The role of HttpClient mentioned above is reflected, and it is used to initialize the key to our sending request: HttpClientRequest. So now the overall logic of our sending request is probably clear.

to sum up

Whether using RestTemplate or rpc annotations to send requests, the underlying logic is actually the same. That is, first match the service information of the other party according to the request information, and then go through a series of Handler processing, such as current limiting, load, fault tolerance, etc. (this is also a good extension mechanism), and finally go to the TransportClientHandler Handler, and then according to Condition to initialize the sent request, after being processed by HttpClientFilter, it will be delegated to vertx's HttpClientRequest to actually send the request.

Click to follow to learn about Huawei Cloud's fresh technology for the first time~