A method of ServletOutputStream output file download under nginx forwarding

The file downloads we have encountered before are basically read in real time. For example, we directly output some server-side files, or export some excel that is not too computationally intensive, so we did not pay too much attention to the details of file downloads.

Yesterday, I took a data export. Since the amount of exported data is not small, a little bit of operation is needed during the export process, which causes the download to take about 5 minutes. At this time, details that hadn't been noticed before surfaced.

The problem this time mainly occurred when the browser did not pop up the file download box in time, which gave the user an illusion that the download page could not be opened.

File stream

When HTTP is connected, it will receive the response header and response body. According to the description of the HTTP protocol, a blank line is used to separate the response header and the body. When the response content is relatively large, the server first sends the response content to the client from top to bottom. This is more like a queue in a data structure. The header information is sent to the queue, and the body's body information is sent to the queue. Then, due to the limitation of the network speed, there is no way to send all the contents of the queue at one time, so it will be sent at the time of sending. Using the first-in-first-out principle, the information of the header at the head of the team is sent to the client first.

The browser as the client, after receiving the HTTP header information, you can know that the background will send a large file to us, and then a dialog box for saving the file will pop up.

Pop up file download box

Therefore, when downloading large files, if you want the browser to pop up the download dialog box in time, the most important thing is to let the browser receive the relevant header information in time.

Therefore, the following code is wrong:

    ServletOutputStream outputStream = httpServletResponse.getOutputStream();
   // 设置响应头 httpServletResponse.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;");
    httpServletResponse.setHeader("Content-Disposition", "attachment;filename=text.xlsx");

    // 模拟耗时的下载
    Thread.sleep(10000);

    // 发送数据并关闭链接
    outputStream.flush(); ➊
    outputStream.close();

The above code will cause the browser to receive the response header information when it is executed to ➊. In other words: only when the code in the background is executed to ➊, will the browser pop up a dialog box to save the file.

The correct method is as follows:

    ServletOutputStream outputStream = httpServletResponse.getOutputStream();
   // 设置响应头 httpServletResponse.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;");
    httpServletResponse.setHeader("Content-Disposition", "attachment;filename=text.xlsx"
    outputStream.flush(); ➋

    // 耗时的下载
    Thread.sleep(10000);

    // 发送数据并关闭链接
    outputStream.flush();
    outputStream.close();

At this time, when the code is executed to ➋, the browser will receive the necessary header information, and then trigger its pop-up dialog box.

I thought everything was going well, but in the words of many novices: the browser means that the does not pop up a dialog box. In fact, the browser does not pop up a dialog box, we have not it out . If 1618ce2a4547ab does not exist, it means 1618ce2a4547ac. When we ask questions, if is , which often means that our mentality has collapsed.

NGINX

Excluding this is the problem, it is often necessary to simplify the environment, throw away some of the environment layer by layer, and see if the error is still reported. After trying, I found out that it was because I used the nginx reverse proxy that the header information was not received by the browser in time, so I boldly guessed that nginx did the data caching.

Due to the relatively small amount of header data sent, NGINX did not choose to forward the data to the browser in real time for some efficiency reasons. This led to the fact that although the HEADER header information was sent out in the background, it was not browsed. The browser receives it, so the browser does not have a real-time pop-up download dialog box.

After getting the general direction, I found that nginx does have a caching function by default through a google query. And then find the corresponding official document on proxy_buffer section. The content is as follows:

---

Syntax:    proxy_buffering on | off;
Default:    
proxy_buffering on;
Context:    http, server, location

Enables or disables buffering of responses from the proxied server.

When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.

When buffering is disabled, the response is passed to a client synchronously, immediately as it is received. nginx will not try to read the whole response from the proxied server. The maximum size of the data that nginx can receive from the server at a time is set by the proxy_buffer_size directive.

Buffering can also be enabled or disabled by passing “yes” or “no” in the “X-Accel-Buffering” response header field. This capability can be disabled using the proxy_ignore_headers directive.

---

By reading the official documentation, we found that there are two ways to disable the cache:

1. Set the value of proxy_buffering to off
2. When returning to the header, add an item X-Accel-Buffering and set the value to no

Through testing, it is found that both cases can work normally. Considering that nginx's caching mechanism must have its own reason, we use the second method here: when returning the header, add an item X-Accel-Buffering and set the value to no :

    ServletOutputStream outputStream = httpServletResponse.getOutputStream();
    httpServletResponse.setContentType("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=utf-8");
    httpServletResponse.setHeader("Content-Disposition", "attachment;filename=" + URLEncoder.encode(filename, "UTF-8"));
    httpServletResponse.setHeader("X-Accel-Buffering", "no");
    outputStream.flush();

At this point, you can finish work happily.

Summarize

The problem that the dialog box did not pop up in time for downloading large files made me realize the role of the flush() method, and also seemed to understand why there is only a set method in response, but no clear method. At the same time, this makes me boldly guess: once the setHeader method is called after the write() method in the response is called, an error or exception should be reported. At the same time, I learned that nginx automatically enabled caching for some unknown reasons. This should be an excellent mechanism, so we did not turn it off directly in the solution. Instead, I chose another method of customizing the header value. This method tells nginx: The data here does not need to be cached, please send it directly to the client. So as to achieve the purpose of real-time pop-up save file dialog box in the browser under the premise of nginx forwarding.

hope it is of help to you.