HTTP Client Study Notes (1) First Encounter

Because I often call third-party interfaces, HttpClient is one of the frameworks I often use. I intend to study it systematically, and at the same time echo the "HTTP Protocol Study Notes (1) First Encounter", one side is theory, the other is practice. At the same time, I have stayed in JDK8 for a long time, and I plan to learn about the new version of JDK features. I noticed that JDK 11 also has an HTTP Client. In this article, our focus is to build HTTP requests, send requests, and then parse the responses. This article also has a different style.

Start with a mission

The protagonist of our story is Xiao Chen, who is still an intern at the moment. The first requirement when he first entered the company was to schedule a third-party interface to pull data into the specified table. Xiao Chen thought that his previous blog had taught Apache Apache. HttpClient example, so I wrote the code to pull down the data as follows:

 @Override
 public void collectData() {
        StringEntity stringEntity = new StringEntity("", ContentType.APPLICATION_JSON);
        HttpUriRequest request = RequestBuilder.post("").addHeader("请求头", "请求尾").addParameter("key", "value").setEntity(stringEntity).build();
        CloseableHttpClient httpClient = HttpClients.createDefault();
        try {
            CloseableHttpResponse response = httpClient.execute(request);
            // 拿到响应
            String responseString = EntityUtils.toString(response.getEntity());
            // 假设拿到了需要调用对面接口的次数
            for (int i = 0; i < Integer.parseInt(responseString) ; i++) {
                 httpClient = HttpClients.createDefault();
                 response = httpClient.execute(request);
                 // 做对应的业务处理
            }
        } catch (IOException e) {
            // 日志暂时省略。
        }
   }

Xiao Chen felt that he had completed this requirement, so he went to the leader to look at the code. After all, he was an intern, and the company still had to control the quality of the code. After the leader read it, the following dialogue occurred:

Leader: Xiao Chen, which protocol is the HTTP protocol based on at the application layer?

Xiao Chen thought that I was familiar with this, and I had memorized this interview. I worked hard on the three-way handshake and four-way handshake, so I answered: HTTP is based on TCP.

The leader then asked: So how does this HttpClient use the TCP protocol?

Xiao Chen is secretly happy, but fortunately, I have paid attention to the technical teenager who loves soup on the public account, and watched the technical teenager who loves soup on the public account "TCP Study Notes (1) First Encounter", "Introduction to Computer Networks" ",then

Answer: The TCP/IP protocol already resides in the modern operating system. In Java, the Apache HttpClient mainly implements the TCP/IP protocol family of the operating system by calling Java to provide Socket-related classes.

When network communication is required, the final call is made to the operating system, the operating system will create a Socket (socket) for the process, and the operating system will allocate relevant resources (storage space, bandwidth, etc.) to the socket.

Leader, at this point, I have to draw a picture to demonstrate my level of networking relevance:

The leader smiled and said: Then if you generate HttpClient in a loop, is there any possibility that when the number of loops is too large, it will take up a lot of Socket resources of the operating system? You see what happens when you keep creating CloseableHttpClient objects?

Xiao Chen thought about it: Yes, then I use it outside the loop?

The leader went on to say: Have you read the documentation of Apache HttpClient? Is it possible for this CloseableHttpClient to be thread-safe?

Xiao Chen suddenly realized and said: Then I will add it to the IOC container, so that our entire system can use this HttpClient, saving resources.

The leader went on to say: Then there are other problems, so think about it.

Xiao Chen said: I can't think of it.

The leader said: Is it possible for the communication process to fail? Assuming that the network is congested at a certain time, is it possible for HttpClient to fail?

Xiao Chen thought about it for a while, and he seemed to be able to, and then asked: Then let's try harder? But how to be more elegant? What I currently think of is to use try catch when calling, if the catch is abnormal, then the for loop will retry the number of times.

The leader smiled and said: Is there such a possibility that this Apache HttpClient has a retry device, go down and check it, and then change it and let's see?

Xiao Chen said: OK.

First, Xiao Chen opened the source code of CloseableHttpClient:

It seems that there is nothing wrong with what the leader said, this class seems to be thread-safe. I thought that the leader mentioned this Apache HttpClient document, so I opened Baidu and entered Apache HttpClient in the search box:

Xiao Chen felt that he was satisfied, so he wrote the following code:

 @Configuration
public class HttpClientConfiguration {
    @Bean
    public CloseableHttpClient closeableHttpClient(){
        HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {
            @Override
            public boolean retryRequest(
                    IOException exception,
                    int executionCount,
                    HttpContext context) {
                if (executionCount >= 5) {
                    // Do not retry if over max retry count
                    return false;
                }
                if (exception instanceof InterruptedIOException) {
                    // Timeout
                    return false;
                }
                if (exception instanceof UnknownHostException) {
                    // Unknown host
                    return false;
                }
                if (exception instanceof ConnectTimeoutException) {
                    // Connection refused
                    return false;
                }
                if (exception instanceof SSLException) {
                    // SSL handshake exception
                    return false;
                }
                HttpClientContext clientContext = HttpClientContext.adapt(context);
                HttpRequest request = clientContext.getRequest();
                // idempotent 幂等
                boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
                if (idempotent) {
                    // Retry if the request is considered idempotent
                    return true;
                }
                return false;
            }

        };
        CloseableHttpClient httpClient = HttpClients.custom().setRetryHandler(myRetryHandler).build();
        return httpClient;
    }
}
 @Override
 public void collectData() {
        StringEntity stringEntity = new StringEntity("", ContentType.APPLICATION_JSON);
        HttpUriRequest request = RequestBuilder.post("").addHeader("请求头key", "请求value").addParameter("key", "value").setEntity(stringEntity).build();
        try {
            CloseableHttpResponse response = closeableHttpClient.execute(request);
            // 拿到响应
            String responseString = EntityUtils.toString(response.getEntity());
            // 假设拿到了需要调用对面接口的次数
            for (int i = 0; i < Integer.parseInt(responseString); i++) {
                closeableHttpClient = HttpClients.createDefault();
                response = closeableHttpClient.execute(request);
                // 做对应的业务处理
            }
        } catch (IOException e) {

        }
   }

Then I called the leader over and asked: Boss, review my code again.

The leader nodded and said: Try again, it's all done, it's ok. The next question, in the case of too many calls, will this CloseableHttpClient open up a TCP connection for each HttpClient connection? If so, is it a bit extravagant, so let's stop looking at the source code.

Xiao Chen thought about it and said: Is it the keep-alive of HTTP1.1?

The leader smiled and said: Yes, there is actually another problem. If the interface is requested four or five thousand times, is this for loop a bit time-consuming?

Xiao Chen suddenly realized: Then here I will open a thread pool to do it, HttpClient will be a connection pool, so that it can be reused like a database connection pool?

The leader nodded and said: Then you can change it again.

So Xiao Chen came to the official website of Apache Httpclient again:

Seeing that the problem of long connections here is solved, what about the connection pool, my God, I won't let me write a connection pool myself, Xiao Chen thought that there must be something in Apache HttpClient, so he continued to read the document:

Then the final program is changed to look like this:

 @Configuration
public class HttpClientConfiguration {
    @Bean
    public CloseableHttpClient closeableHttpClient(){
        HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler() {

            @Override
            public boolean retryRequest(
                    IOException exception,
                    int executionCount,
                    HttpContext context) {
                // 返回true 就代表重试
                if (executionCount >= 5) {
                    // Do not retry if over max retry count
                    return false;
                }
                if (exception instanceof InterruptedIOException) {
                    // Timeout
                    return false;
                }
                if (exception instanceof UnknownHostException) {
                    // Unknown host
                    return false;
                }
                if (exception instanceof ConnectTimeoutException) {
                    // Connection refused
                    return false;
                }
                if (exception instanceof SSLException) {
                    // SSL handshake exception
                    return false;
                }
                HttpClientContext clientContext = HttpClientContext.adapt(context);
                HttpRequest request = clientContext.getRequest();
                boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
                if (idempotent) {
                    // Retry if the request is considered idempotent
                    return true;
                }
                return false;
            }

        };
        PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
        // Increase max total connection to 200
        cm.setMaxTotal(200);
        // Increase default max connection per route to 20
        cm.setDefaultMaxPerRoute(20);
        // Increase max connections for localhost:80 to 50
        HttpHost localhost = new HttpHost("locahost", 80);
        cm.setMaxPerRoute(new HttpRoute(localhost), 50);
        CloseableHttpClient httpClient = HttpClients.custom().setRetryHandler(myRetryHandler).setConnectionManager(cm).build();
        return httpClient;
    }
}

Regarding multiple requests, Xiao Chen intends to make a thread pool to process requests concurrently, so the program eventually becomes the following:

 @Override
 public void collectData() {
        StringEntity stringEntity = new StringEntity("", ContentType.APPLICATION_JSON);
        HttpUriRequest request = RequestBuilder.post("").addHeader("请求头", "请求尾").addParameter("key", "value").setEntity(stringEntity).build();
        try {
            CloseableHttpResponse response = closeableHttpClient.execute(request);
            // 拿到响应
            String responseString = EntityUtils.toString(response.getEntity());
            // 假设拿到了需要调用对面接口的次数
            for (int i = 0; i < Integer.parseInt(responseString); i++) {
                POOL_EXECUTOR.submit(()->{
                    try {
                        CloseableHttpResponse threadResponseString = closeableHttpClient.execute(request);
                    } catch (IOException e) {
                        
                    }
                });
               
                // 做对应的业务处理
            }
        } catch (IOException e) {

        }
  }

So Xiao Chen found the leader again and asked him to review his code again. The leader looked at it and nodded: OK, barely passed, this task is considered to be completed by you, then let's learn more about HttpClient in JDK 11 Well, this time your task is to have a general understanding of the new HttpClient in JDK 11, and you can basically use it. The main purpose is to let you see the implementation of different HttpClients.

HTTP Client for JDK 11

Xiao Chen got the task assigned by the leader. First, search for open jdk on Baidu. Open jdk will have a description of the new features of JDK 11:

The following is a description of this feature in JDK 11

The existing HttpURLConnection API and its implementation have numerous problems:

• The base URLConnection API was designed with multiple protocols in mind, nearly all of which are now defunct ( ftp , gopher , etc.).
• The API predates HTTP/1.1 and is too abstract.
• It is hard to use, with many undocumented behaviors.
• It works in blocking mode only (ie, one thread per request/response).
• It is very hard to maintain.

There are many problems with the existing HttpURLConnection API implementation:

• URLConnection is designed for a variety of protocols, but most of those protocols don't exist anymore
• This interface predates HTTP Client, but is too abstract.
• Difficult to use, and some behaviors are not commented
• Only blocking mode (one thread for each pair of request and response)

After reading this description, Xiao Chen's first thought was, how should I use JDK 11's HttpClient.

JDK 11 introduces the new HttpClient like this:

The HTTP Client was added in Java 11. It can be used to request HTTP resources over the network. It supports HTTP/1.1 and HTTP/2 , both synchronous and asynchronous programming models, handles request and response bodies as reactive-streams, and follows the familiar builder pattern.

This new implementation of HTTP Client was introduced in Java 11 and can be used in the network to request HTTP resources, supporting HTTP1.1, HTTP/2, synchronous and asynchronous modes. Handling request and response streams supports the response stream pattern and can also be constructed in a familiar way.

• Example 1 Interpretation:

 public static void main(String[] args) {
        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("http://openjdk.java.net/"))
                .build();
        // 这是一个异步请求
        // 处理响应将响应当作一个字符串,
        // 返回结果CompletableFuture对象,不知道CompletableFuture
         // 可以去翻一下我之前写的《Java多线程学习笔记(六) 长乐未央篇》
         // thenApply 收到线程的时候 消费HttpResponse的数据,最终被
         // thenAccept所处理,join同Thread.join方法一样,
         //CompletableFuture链式处理数据完毕才会走到下一行
        client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
                .thenApply(HttpResponse::body)
                .thenAccept(System.out::println).join();
    }
}

Didn't the above say that HTTP/1.1 and HTTP/2 are supported, so how do I use it, and how do I add request parameters?

 // 通过Version字段指定
HttpClient client = HttpClient.newBuilder()
      .version(Version.HTTP_2)

The construction of Http requests is mainly based on HttpRequest.newBuilder(), and newBuilder finally points to HttpRequestBuilderImpl. Let's see what parameters the HttpRequestBuilderImpl class can help us build:

 HttpRequest request = HttpRequest.newBuilder()
      .uri(URI.create("http://openjdk.java.net/")) // URI 是地址
      .timeout(Duration.ofMinutes(1)) // 超时时间
      .header("Content-Type", "application/json") // 请求头
      .POST(BodyPublishers.ofFile(Paths.get("file.json"))) // 参数主要通过BodyPublishers来构建
      .build()

Seeing this, Xiao Chen is thinking, why does JDK 11's HttpClient have no addParameter when building a request for Apache HttpClient? I haven't found the paramsPushlisher operation in a lot of information, but it seems that we need to splicing it on the URL ourselves. Building the request body HttpClient provides us with BodyPublishers to build:

 HttpRequest.BodyPublishers::ofByteArray(byte[]) // 向服务端发送字节数组
HttpRequest.BodyPublishers::ofByteArrays(Iterable)
HttpRequest.BodyPublishers::ofFile(Path) // 发送文件
HttpRequest.BodyPublishers::ofString(String) // 发送String 
HttpRequest.BodyPublishers::ofInputStream(Supplier<InputStream>) // 发送流

Send a POST request:

 // HttpResponse.BodyHandlers 用来处理响应体中的数据,ofString,将响应体中的数据处理成String
client.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
thenAccept(System.out::println).join();

So far, I haven't found out how JDK 11's HttpClient manages connections. Someone asked this question on StackOverFlow, which is the default policy of JDK. But if some requests do not need keep-alive, it seems that this has to be achieved by changing the properties of the entire JVM, and how to implement retry, which is not found here. Retry is still very important in some scenarios. If you use the one that comes with JDK 11, you need to repackage it yourself. Apache HttpClient is also talked about under this JEP:

 A number of existing HTTP client APIs and implementations exist, e.g., Jetty and the Apache HttpClient. Both of these are both rather heavy-weight in terms of the numbers of packages and classes, and they don't take advantage of newer language features such as lambda expressions.
已经存在了一些HTTP Client库，Jetty和Apache HttpClient，就代码量来说这两个都相当的庞大，他们也没有用到JDK的新特性比如Lamda表达式。

write at the end

This article basically introduces the basic use of Apache HttpClient and JDK 11's Httpclient. At present, it seems that Apache HttpClient is more complete, but the implementation of JDK 11 also has bright spots, such as the encapsulation of the response is also very exciting .

References

• How to keep connection alive in Java 11 http client https://stackoverflow.com/questions/53617574/how-to-keep-connection-alive-in-java-11-http-client
• Introduction to the Java HTTP Client https://openjdk.java.net/groups/net/httpclient/intro.html