Netty series: build your own download file server

Introduction

In the previous article, we learned how to build an HTTP server in netty, and discussed how to process and respond to requests sent by the client. Today we will discuss the issues that should be paid attention to when setting up a file server in netty for file transfer.

The content-type of the file

The client requests a file from the server, and the server will include a content-type in the returned HTTP header. This content-type represents the type of the returned file. How should this type be confirmed?

Generally speaking, the file type is confirmed according to the file extension. According to the RFC 4288 specification, all network media types must be registered. Apache also provides a mapping table of file MIME type and extension.

Because there are many file types, let's look at a few of the more commonly used types as follows:

MIME type	extension name
image/jpeg	jpg
image/jpeg	jpeg
image/png	png
text/plain	txt text conf def list log in
image/webp	webp
application/vnd.ms-excel	xls
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	xlsx
application/msword	doc
application/vnd.openxmlformats-officedocument.wordprocessingml.document	docx
application/vnd.openxmlformats-officedocument.presentationml.presentation	pptx
application/vnd.ms-powerpoint	ppt
application/pdf	pdf

JDK provides a class of MimetypesFileTypeMap. This class provides a getContentType method, which can infer the MIME type based on the requested file path information:

    private static void setContentTypeHeader(HttpResponse response, File file) {
        MimetypesFileTypeMap mimeTypesMap = new MimetypesFileTypeMap();
        response.headers().set(HttpHeaderNames.CONTENT_TYPE, mimeTypesMap.getContentType(file.getPath()));
    }

Client cache file

For HTTP file requests, in order to ensure the speed of the request, a client-side caching mechanism is used. For example, the client requests a file A.txt from the server. The server will send the A.txt file to the client after receiving the request.

The request process is as follows:

   步骤1：客户端请求服务器端的文件
   ===================
   GET /file1.txt HTTP/1.1

   步骤2：服务器端返回文件，并且附带额外的文件时间信息：
   ===================
   HTTP/1.1 200 OK
   Date:               Mon, 23 Aug 2021 17:52:30 GMT+08:00
   Last-Modified:      Tue, 10 Aug 2021 18:05:35 GMT+08:00
   Expires:            Mon, 23 Aug 2021 17:53:30 GMT+08:00
   Cache-Control:      private, max-age=60

Generally speaking, if the client is a modern browser, A.txt will be cached. In the next call, you only need to add If-Modified-Since to the head and ask the server whether the file has been modified. If the file has not been modified, the server will return a 304 Not Modified. After the client gets the status, The local cache file will be used.

   步骤3：客户端再次请求该文件
   ===================
   GET /file1.txt HTTP/1.1
   If-Modified-Since:  Mon, 23 Aug 2021 17:55:30 GMT+08:00
  
   步骤4：服务器端响应该请求
   ===================
   HTTP/1.1 304 Not Modified
   Date:               Mon, 23 Aug 2021 17:55:32 GMT+08:00

At the code level of the server, we first need to return a date field that is usually required in the response, such as Date, Last-Modified, Expires, Cache-Control, etc.:

 SimpleDateFormat dateFormatter = new SimpleDateFormat(HTTP_DATE_FORMAT, Locale.US);
        dateFormatter.setTimeZone(TimeZone.getTimeZone(HTTP_DATE_GMT_TIMEZONE));

        // 日期 header
        Calendar time = new GregorianCalendar();
        log.info(dateFormatter.format(time.getTime()));

        response.headers().set(HttpHeaderNames.DATE, dateFormatter.format(time.getTime()));

        // 缓存 headers
        time.add(Calendar.SECOND, HTTP_CACHE_SECONDS);
        response.headers().set(HttpHeaderNames.EXPIRES, dateFormatter.format(time.getTime()));
        response.headers().set(HttpHeaderNames.CACHE_CONTROL, "private, max-age=" + HTTP_CACHE_SECONDS);
        response.headers().set(
                HttpHeaderNames.LAST_MODIFIED, dateFormatter.format(new Date(fileToCache.lastModified())));

Then after receiving the second request from the client, you need to compare the last modification time of the file with the time in If-Modified-Since. If no changes are sent, send the 304 status:

FullHttpResponse response = new DefaultFullHttpResponse(HTTP_1_1, NOT_MODIFIED, Unpooled.EMPTY_BUFFER);
        setDateHeader(response);

Other common processing in HTTP

We discussed file types and caching. For a general HTTP server, there are many other common handlings that need to be considered, such as exceptions, redirects, and Keep-Alive settings.

For exceptions, we need to construct a DefaultFullHttpResponse based on the exception code, and set the corresponding CONTENT_TYPE header, as shown below:

FullHttpResponse response = new DefaultFullHttpResponse(
                HTTP_1_1, status, Unpooled.copiedBuffer("异常: " + status + "\r\n", CharsetUtil.UTF_8));
        response.headers().set(HttpHeaderNames.CONTENT_TYPE, "text/plain; charset=UTF-8");

Redirection also needs to construct a DefaultFullHttpResponse whose status is 302 Found, and set location as the URL address to be redirected in the response header:

FullHttpResponse response = new DefaultFullHttpResponse(HTTP_1_1, FOUND, Unpooled.EMPTY_BUFFER);
        response.headers().set(HttpHeaderNames.LOCATION, newUri);

Keep-Alive is an optimization method in HTTP to avoid establishing a connection for each request. The default keep-alive in HTTP/1.0 is false, and the default keep-alive in HTTP/1.1 is true. If connection: false is manually set in the header, the server-side request return also needs to set connection: false.

In addition, because the default keep-alive in HTTP/1.1 is true, if it is passed through HttpUtil.isKeepAlive, it is necessary to determine whether it is HTTP/1.0 and display the keep-alive setting as true.

final boolean keepAlive = HttpUtil.isKeepAlive(request);
        HttpUtil.setContentLength(response, response.content().readableBytes());
        if (!keepAlive) {
            response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);
        } else if (request.protocolVersion().equals(HTTP_1_0)) {
            response.headers().set(HttpHeaderNames.CONNECTION, HttpHeaderValues.KEEP_ALIVE);
        }

File content display processing

File content display processing is the core of the http server, and it is also difficult to understand.

The first thing to set is ContentLength, which is the length of the response file. This can be obtained using the length method of file:

RandomAccessFile raf；
raf = new RandomAccessFile(file, "r");
long fileLength = raf.length();
HttpUtil.setContentLength(response, fileLength);

Then we need to set the corresponding CONTENT_TYPE according to the file extension, which has been introduced in the first section.

Then set the date and cache attributes. In this way, we get a DefaultHttpResponse that only contains the response header, and we first write this response that only contains the response header to ctx.

After writing the HTTP header, the next step is to write the HTTP Content.

There are two processing methods for files transmitted by HTTP. In the first method, if the content size of the entire response is known, the entire file can be copied and transmitted directly in the background. If the server itself supports zero copy, you can use the transferTo method of DefaultFileRegion to transfer File or Channel files.

sendFileFuture =
                    ctx.write(new DefaultFileRegion(raf.getChannel(), 0, fileLength), ctx.newProgressivePromise());
            // 结束部分
            lastContentFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);

If you don’t know the context size of the entire response, you can split large files into chunks, and set transfer-coding to chunked in the response header. Netty provides HttpChunkedInput and ChunkedFile to split large files Become a Chunk for transmission.

sendFileFuture =
                    ctx.writeAndFlush(new HttpChunkedInput(new ChunkedFile(raf, 0, fileLength, 8192)),
                            ctx.newProgressivePromise());

If you write ChunkedFile to the channel, you need to add the corresponding ChunkedWriteHandler to process the chunked file.

pipeline.addLast(new ChunkedWriteHandler());

Note that if it is a complete file transfer, you need to manually add the last content part:

lastContentFuture = ctx.writeAndFlush(LastHttpContent.EMPTY_LAST_CONTENT);

If it is ChunkedFile, the last content part is already included in chunkedFile, and there is no need to add it manually.

File transfer progress

ChannelFuture can add a corresponding listner to monitor the progress of file transmission. Netty provides a ChannelProgressiveFutureListener to monitor the progress of files. You can override the operationProgressed and operationComplete methods to customize the progress monitoring:

        sendFileFuture.addListener(new ChannelProgressiveFutureListener() {
            @Override
            public void operationProgressed(ChannelProgressiveFuture future, long progress, long total) {
                if (total < 0) {
                    log.info(future.channel() + " 传输进度: " + progress);
                } else {
                    log.info(future.channel() + " 传输进度: " + progress + " / " + total);
                }
            }

            @Override
            public void operationComplete(ChannelProgressiveFuture future) {
                log.info(future.channel() + " 传输完毕.");
            }
        });

Summarize

We have considered some of the most basic considerations of an HTTP file server, and now you can use this file server to provide services!

For the examples in this article, please refer to: learn-netty4

This article has been included in http://www.flydean.com/20-netty-fileserver/

The most popular interpretation, the most profound dry goods, the most concise tutorial, and many tips you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", know technology, know you better!