How to locate and fix HTTP2 flow control issues in HttpCore5

Author: Fengqi

At the beginning, I will blow a wave of Alibaba Cloud performance testing service PTS [1] . PTS has launched support for the HTTP2 protocol in May 2021 (the bottom layer relies on httpclient5), which will be determined by the results of negotiation with the server during stress testing. Use HTTP1.1 or HTTP2 protocol.

background

The reason for writing this article is that a customer came over one day and asked us if we did not support HTTP2, because he purchased 2 domain names on XX Cloud, one of which enabled HTTP2, and during the PTS stress test process, support The HTTP2 interface always reports an error:

At first, it was suspected to be a problem with HTTP2 support. By forcing the use of the HTTP2 protocol locally and accessing the Taobao homepage, it was found that there was no problem. It was suspected that it was a configuration problem of the user on the XX cloud, but then through the local Postman, curl and pressure test engine When the HTTP1.1 protocol is forced to be used, the webpage can be accessed normally, and I realize that there is a high probability that it is a problem on the PTS engine side.

Through local debug, it is seen that the client window size is adjusted to be larger than 2^32 -1 when requesting the URL.

Then just take this opportunity to see what the window size here refers to.

HTTP2 flow control

When it comes to windows, it is necessary to mention a new feature supported by HTTP2 compared to HTTP1.1: Flow Control. In fact, HTTP1.1 relies on the sliding window of the transport layer TCP to achieve flow control, so why does HTTP2 need to be used in What about implementing a flow control at the application layer? The reason is that HTTP2 introduces streams and multiplexing, and the effect of coordinating multiple streams can be achieved through flow control.

Some basic concepts of flow control:

Flow control is for the connection, not for end-to-end, but at each hop at both ends; mainly refers to the case where there is a proxy, there is flow control between the proxy and both ends
Flow control is based on the WINDOW_UPDATE frame, and the receiver can control the sender's speed through flow control
Flow control can act on both stream and connection
For connections and all newly opened streams, the default flow control window size is 65535, and the maximum value is 2^32 - 1
Flow control cannot be disabled

For ease of understanding, let's briefly list the types of HTTP2 frames:

DATA: carry the data in the request or response
HEADERS: used to create a new stream (request or response), including the corresponding Headers
PRIORITY: Used to configure the priority of the stream
RST_STREAM: Forcibly end a stream, only used to cancel the stream at one end, not applicable to the end of a normal stream
SETTINGS: Some configurations for H2 connection
PUSH_PROMISE: The server pushes the response to the client
PING: Send a PING to the remote end, and the remote end must return the PING
GOAWAY: Used for a connection that is about to end at one end
WINDOW_UPDATE: Update flow control window size
CONTINUATION: If the headers are too large and it is difficult to carry a single HEADERS frame, send additional headers through this frame

Next, let's focus on the frames related to flow control, mainly SETTING and WINDOW_UPDATE. When the connection is established, the window size of the other party will be adjusted through the SETTINGS frame, and then during the transmission process, the window size will gradually decrease with the transmission of data , until the WINDOW_UPDATE frame sent by the other party is received, thereby updating the window size. The SETTINGS frame mainly contains the following:

SETTINGS_HEADER_TABLE_SIZE: HPACK (a header compression algorithm) the maximum length of the header table, the default value is 4096
SETTINGS_ENABLE_PUSH: The configuration sent by the client to the server. If set to true, the client will allow the server to push the response. The default value is true
SETTINGS_MAX_CONCURRENT_STREAMS: The maximum number of streams opened at the same time, usually means the number of requests that can be responded to at the same time, the default is unlimited
SETTINGS_INITIAL_WINDOW_SIZE: The initial window size for flow control, the default value is 65535
SETTINGS_MAX_FRAME_SIZE: The maximum length of the frame that the peer can accept, the default value is 16384
SETTINGS_MAX_HEADER_LIST_SIZE: The maximum length of the header list that the peer can accept, no limit by default

The implementation of flow control is as described above, every time a batch of DATA frames is sent, the window size is reduced. Note that flow control is only for DATA frames.

As mentioned earlier, flow control can act on both stream and connection. How is it implemented? The flow control of connection is similar to the above stream flow control logic. Each time a DATA frame is sent, the connection and stream windows will be reduced, but the difference is that WINDOW_UPDATE either acts on the stream alone, or acts on the connection alone (when the streamid is 0, it means acting on the connection).

identify the problem

So back to the question of the opening, we have the URL of https://www.sysgeek.cn/ for example, by making the code debug locally found the ultimate reason for throwing the exception that received WINDOW_UPDATE frames, window after update Size values greater than 2^32 - 1 cause an exception to be thrown:

As can be seen from the code here, 524288 is the current window size, and delta is the size of WINDOW_UPDATE notified by the other party. Through analysis, it is found that the value of 524288 is different from the default value of 65535. Then continue to see when this value was changed:

It is found that the window size is modified after receiving the SETTINGS command, but this conflicts with [2] (the connection window size may only be modified after receiving WINDOW_UPDATE):

Therefore, we concluded that there is a bug in the source code of httpcore5. After deleting the marked line of code, the request can be executed normally.

Unfortunately, in the process of preparing to submit a PR to httpcore5, I found that this bug has been fixed in the commit.