Netty series: build HTTP upload file server

Introduction

In the last article, we talked about how to download files from the HTTP server, and the problems that should be paid attention to when building a download file server, and the GET method used. This article will discuss the commonly used POST method to submit data to the server and how to upload files to the server.

GET method to upload data

According to the HTTP specification, PUT is generally to upload data to the server. Although it is not recommended, you can also use GET to upload data to the server.

Let's first look at the issues that need to be paid attention to in the construction of the GET client.

A GET request is actually a URI with the requested parameters behind the URI. Netty provides a QueryStringEncoder specifically to construct the parameter content:

// HTTP请求
        QueryStringEncoder encoder = new QueryStringEncoder(get);
        // 添加请求参数
        encoder.addParam("method", "GET");
        encoder.addParam("name", "flydean");
        encoder.addParam("site", "www.flydean.com");
        URI uriGet = new URI(encoder.toString());

With the request URI, you can create an HttpRequest. Of course, this HttpRequest also needs the corresponding HTTP head data:

HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, uriGet.toASCIIString());
        HttpHeaders headers = request.headers();
        headers.set(HttpHeaderNames.HOST, host);
        headers.set(HttpHeaderNames.CONNECTION, HttpHeaderValues.CLOSE);
        headers.set(HttpHeaderNames.ACCEPT_ENCODING, HttpHeaderValues.GZIP + "," + HttpHeaderValues.DEFLATE);
        headers.set(HttpHeaderNames.ACCEPT_LANGUAGE, "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2");
        headers.set(HttpHeaderNames.REFERER, uriSimple.toString());
        headers.set(HttpHeaderNames.USER_AGENT, "Netty Simple Http Client side");
        headers.set(HttpHeaderNames.ACCEPT, "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");

        headers.set(
                HttpHeaderNames.COOKIE, ClientCookieEncoder.STRICT.encode(
                        new DefaultCookie("name", "flydean"),
                        new DefaultCookie("site", "www.flydean.com"))
        );

We know that there are only two parts of data in HttpRequest, namely HttpVersion and HttpHeaders. HttpVersion is the version number of the HTTP protocol, and HttpHeaders is the set header content.

For GET requests, because all the content is contained in the URI, no additional HTTPContent is needed, just send the HttpRequest directly to the server.

channel.writeAndFlush(request);

Then look at how to process the server after receiving the GET request.

After the server receives the msg of the HttpObject object, it needs to be converted into an HttpRequest object, and the corresponding information can be obtained through protocolVersion(), uri() and headers().

For the parameters in the URI, netty provides the QueryStringDecoder class to easily parse the parameters in the URI:

//解析URL中的参数
            QueryStringDecoder decoderQuery = new QueryStringDecoder(request.uri());
            Map<String, List<String>> uriAttributes = decoderQuery.parameters();
            for (Entry<String, List<String>> attr: uriAttributes.entrySet()) {
                for (String attrVal: attr.getValue()) {
                    responseContent.append("URI: ").append(attr.getKey()).append('=').append(attrVal).append("\r\n");
                }
            }

POST method to upload data

For POST requests, it has one more HTTPContent than GET requests, which means that in addition to the basic HttpRequest data, a PostBody is also required.

If it is just an ordinary POST, that is, the POST content is in the form of key=value, it is relatively simple. If the POST contains files, it will be more complicated, and ENCTYPE="multipart/form-data" needs to be used.

netty provides a HttpPostRequestEncoder class, which is used to quickly encode the request body. First, let's take a look at the complete constructor of the HttpPostRequestEncoder class:

public HttpPostRequestEncoder(
            HttpDataFactory factory, HttpRequest request, boolean multipart, Charset charset,
            EncoderMode encoderMode)

The request is the HttpRequest to be encoded, multipart indicates whether it is in the "multipart/form-data" format, and the charset encoding method is CharsetUtil.UTF_8 by default. EncoderMode is the encoding mode. There are currently three encoding modes, namely RFC1738, RFC3986 and HTML5.

The default encoding mode is RFC1738, which is also the encoding method for most form submission data. But it does not apply to OAUTH, if you want to use OAUTH, you can use RFC3986. HTML5 disables the mixed mode of multipart/form-data.

Finally, we talk about HttpDataFactory. The factory is mainly used to create InterfaceHttpData. It has a minSize parameter. If the size of the created HttpData is larger than minSize, it will be stored in disk, otherwise it will be created directly in memory.

InterfaceHttpData has three types of HttpData, namely Attribute, FileUpload and InternalAttribute.

Attribute is the attribute value passed in in the POST request. FileUpload is the file passed in in the POST request, and InternalAttribute is used inside the encoder, so I won't discuss it here.

Therefore, according to the input minSize parameter size, Attribute and FileUpload can be divided into the following types:

MemoryAttribute, DiskAttribute or MixedAttribute
MemoryFileUpload, DiskFileUpload or MixedFileUpload

In this section, let's take a look at the processing method of not uploading a file in a POST request. First, create an HTTP request and PostBody encoder:

// 构建HTTP request
        HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, uriSimple.toASCIIString());

 HttpPostRequestEncoder bodyRequestEncoder =
                new HttpPostRequestEncoder(factory, request, false);  

Add headers to the request:

// 添加headers
        for (Entry<String, String> entry : headers) {
            request.headers().set(entry.getKey(), entry.getValue());
        }

Then add the form attribute to bodyRequestEncoder:

// 添加form属性
        bodyRequestEncoder.addBodyAttribute("method", "POST");
        bodyRequestEncoder.addBodyAttribute("name", "flydean");
        bodyRequestEncoder.addBodyAttribute("site", "www.flydean.com");
        bodyRequestEncoder.addBodyFileUpload("myfile", file, "application/x-zip-compressed", false);

Note that above we added the method, name and site attributes to the bodyRequestEncoder. Then added a FileUpload. But because our encoding method is not "multipart/form-data", only the file name is passed here, not the entire file.

Finally, we have to call the finalizeRequest method of bodyRequestEncoder to return the final request to be sent. In the process of finalizeRequest, whether transfer-encoding is chunked is also set according to the size of the transferred data.

If the content of the transmission is relatively large, it needs to be transmitted in segments. At this time, you need to set transfer-encoding = chunked, otherwise, do not set it.

Finally send the request:

// 发送请求
        channel.write(request);

On the server side, we also need to construct an HttpDataFactory, and then use this factory to construct an HttpPostRequestDecoder to decode the data from the encoder:

HttpDataFactory factory =
            new DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE);
//POST请求
decoder = new HttpPostRequestDecoder(factory, request);

Because the message received by the server may be HttpContent or LastHttpContent depending on the length of the sent message. If it is HttpContent, we will put the parsed result in a StringBuilder and cache it, and then send it out together after receiving the LastHttpContent.

After receiving the HttpContent, we call the decoder.offer method to decode the HttpContent:

decoder.offer(chunk);

There are two containers for storing HttpData data inside the decoder, which are:

List<InterfaceHttpData> bodyListHttpData
和
Map<String, List<InterfaceHttpData>> bodyMapHttpData

decoder.offer is to parse the chunk, and then fill the parsed data into bodyListHttpData and bodyMapHttpData.

After parsing, you can read the parsed data.

The bodyListHttpData can be traversed through the hasNext and next methods of the decoder to obtain the corresponding InterfaceHttpData.

Through data.getHttpDataType(), you can get the data type of InterfaceHttpData. As mentioned above, there are two types of Attribute and FileUpload.

POST method to upload files

If you want a POST file, the client can pass in multipart=true when creating the HttpPostRequestEncoder:

 HttpPostRequestEncoder bodyRequestEncoder =
                new HttpPostRequestEncoder(factory, request, true);

Then call the setBodyHttpDatas and finalizeRequest methods respectively to generate an HttpRequest and then write to the channel:

// 添加body http data
        bodyRequestEncoder.setBodyHttpDatas(bodylist);
        // finalize request，判断是否需要chunk
        request = bodyRequestEncoder.finalizeRequest();
        // 发送请求头
        channel.write(request);

It should be noted that if transfer-encoding = chunked, then this HttpRequest is just the information of the request header, and we also need to manually write HttpContent to the channel:

        // 判断bodyRequestEncoder是否是Chunked，发送请求内容
        if (bodyRequestEncoder.isChunked()) {
            channel.write(bodyRequestEncoder);
        }

On the server side, by judging the getHttpDataType of InterfaceHttpData, if it is of the FileUpload type, it means that the uploaded file is obtained, and the content of the file can be read by the following method:

FileUpload fileUpload = (FileUpload) data;
responseContent.append(fileUpload.getString(fileUpload.getCharset()));

In this way, we can get the files from the client on the server side.

Summarize

There are many issues that need to be considered for HTTP file upload. If you don't understand, you can refer to my example. Or leave a message for me to discuss together.

For the examples in this article, please refer to: learn-netty4

This article has been included in http://www.flydean.com/21-netty-http-fileupload/

The most popular interpretation, the most profound dry goods, the most concise tutorial, and many tips you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", know technology, know you better!