Netty series: frame decoder in netty

Introduction

The data in netty is transmitted through ByteBuf. A ByteBuf may contain multiple meaningful data, which can be called frames, which means that a ByteBuf can contain multiple Frames.

For the receiver of the message, after receiving the ByteBuf, it is necessary to parse the useful data from the ByteBuf, and then the frame in the ByteBuf needs to be split and parsed.

Generally speaking, there will be some specific delimiters between different frames, and we can use these delimiters to distinguish frames, so as to realize the parsing of data.

Netty provides us with some suitable frame decoders, which can effectively simplify our work by using these frame decoders. The following figure shows several common frame decoders in netty:

<img src="https://img-blog.csdnimg.cn/6f394018c43a40a6a53a5260fc577575.png" style="zoom:67%;" />

Next, let's introduce the use of the above frame decoders in detail.

LineBasedFrameDecoder

LineBasedFrameDecoder from the name point of view is to distinguish the frame by line. Depending on the operating system, a newline can have two newline characters, "\n" and "\r\n".

The basic principle of LineBasedFrameDecoder is to read the corresponding characters from ByteBuf to compare with "\n" and "\r\n", which can accurately compare characters. These frameDecoders also have certain requirements for character encoding. Generally speaking, UTF-8 encoding is required. Because in this encoding, "\n" and "\r" appear as one byte and will not be used in other combined encodings, so it is very difficult to use "\n" and "\r" to judge safe.

There are several important properties in LineBasedFrameDecoder. One is the maxLength property, which is used to detect the length of the received message. If the length limit is exceeded, a TooLongFrameException exception will be thrown.

There is also a stripDelimiter property to determine whether the delimiter needs to be filtered out.

Another is failFast, if the value is true, then regardless of whether the frame is read or not, as long as the length of the frame exceeds maxFrameLength, TooLongFrameException will be thrown. If the value is false, TooLongFrameException will be thrown after the entire frame has been fully read.

The core logic of LineBasedFrameDecoder is to first find the position of the line separator, and then read the corresponding frame information according to this position. Here's a look at the findEndOfLine method to find the line separator:

 private int findEndOfLine(final ByteBuf buffer) {
        int totalLength = buffer.readableBytes();
        int i = buffer.forEachByte(buffer.readerIndex() + offset, totalLength - offset, ByteProcessor.FIND_LF);
        if (i >= 0) {
            offset = 0;
            if (i > 0 && buffer.getByte(i - 1) == '\r') {
                i--;
            }
        } else {
            offset = totalLength;
        }
        return i;
    }

Here, a ByteBuf's forEachByte is used to traverse the ByteBuf. The character we are looking for is: ByteProcessor.FIND_LF.

Finally, the object decoded by LineBasedFrameDecoder is still a ByteBuf.

DelimiterBasedFrameDecoder

The LineBasedFrameDecoder mentioned above is only valid for line delimiters. If our frame is divided by other delimiters, the LineBasedFrameDecoder will not work, so netty provides a more general DelimiterBasedFrameDecoder, this frameDecoder can customize the delimiter:

 public class DelimiterBasedFrameDecoder extends ByteToMessageDecoder {

        public DelimiterBasedFrameDecoder(int maxFrameLength, ByteBuf delimiter) {
        this(maxFrameLength, true, delimiter);
    }

The incoming delimiter is a ByteBuf, so the delimiter may be more than one character.

In order to solve this problem, an array of ByteBuf is defined in DelimiterBasedFrameDecoder:

 private final ByteBuf[] delimiters;

    delimiters= delimiter.readableBytes();

The delimiters are obtained by calling the delimiter's readableBytes.

The logic of DelimiterBasedFrameDecoder is similar to that of LineBasedFrameDecoder. It intercepts the data in the buffer by comparing the characters in the buffer, but the DelimiterBasedFrameDecoder can accept multiple delimiters, so its usefulness will be widely based.

FixedLengthFrameDecoder

In addition to performing frame splitting by comparing characters in ByteBuf, there are some other common frame splitting methods, such as distinguishing according to a specific length. Netty provides such a decoder called FixedLengthFrameDecoder.

 public class FixedLengthFrameDecoder extends ByteToMessageDecoder

FixedLengthFrameDecoder is also inherited from ByteToMessageDecoder, its definition is very simple, you can pass in the length of a frame:

 public FixedLengthFrameDecoder(int frameLength) {
        checkPositive(frameLength, "frameLength");
        this.frameLength = frameLength;
    }

Then call the readRetainedSlice method of ByteBuf to read the fixed-length data:

 in.readRetainedSlice(frameLength)

Finally, the read data is returned.

LengthFieldBasedFrameDecoder

There are also some frames that contain a specific length field, which indicates how much readable data is in the ByteBuf. Such frames are called LengthFieldBasedFrame.

A corresponding processing decoder is also provided in netty:

 public class LengthFieldBasedFrameDecoder extends ByteToMessageDecoder

The logic of reading is very simple, first read the length, and then read the data according to the length. To implement this logic, LengthFieldBasedFrameDecoder provides four fields, lengthFieldOffset, lengthFieldLength, lengthAdjustment and initialBytesToStrip.

lengthFieldOffset specifies the starting position of the length field, lengthFieldLength defines the length of the length field, lengthAdjustment adjusts the lengthFieldLength, and initialBytesToStrip indicates whether the length field needs to be removed.

It sounds like it's not very easy to understand, let's give a few examples, the first is the simplest:

 BEFORE DECODE (14 bytes)         AFTER DECODE (14 bytes)
   +--------+----------------+      +--------+----------------+
   | Length | Actual Content |----->| Length | Actual Content |
   | 0x000C | "HELLO, WORLD" |      | 0x000C | "HELLO, WORLD" |
   +--------+----------------+      +--------+----------------+

The message to be encoded has a length field, and the length field is followed by the real data. 0x000C is a hexadecimal, and the data represented is 12, which is the length of the string in "HELLO, WORLD".

Here the values of the 4 properties are:

 lengthFieldOffset   = 0
   lengthFieldLength   = 2
   lengthAdjustment    = 0
   initialBytesToStrip = 0

Indicates that the length field starts from 0, and the length field occupies two bytes, the length does not need to be adjusted, and the field does not need to be adjusted.

Let's look at a more complex example, in which the four attribute values are as follows:

 lengthFieldOffset   = 1  
   lengthFieldLength   = 2
   lengthAdjustment    = 1  
   initialBytesToStrip = 3

The corresponding encoded data is as follows:

 BEFORE DECODE (16 bytes)                       AFTER DECODE (13 bytes)
   +------+--------+------+----------------+      +------+----------------+
   | HDR1 | Length | HDR2 | Actual Content |----->| HDR2 | Actual Content |
   | 0xCA | 0x000C | 0xFE | "HELLO, WORLD" |      | 0xFE | "HELLO, WORLD" |
   +------+--------+------+----------------+      +------+----------------+

In the above example, the length field starts from the 1st byte (the 0th byte is HDR1), the length field occupies 2 bytes, and the length is adjusted by one byte. The starting position of the final data is 1+2+ 1=4, and then intercept the first 3 bytes of data to get the final result.

Summarize

These character set-based frame decoders provided by netty can basically meet our daily work needs. Of course, if you are transmitting some more complex objects, then you can consider custom encoding and decoding. The custom logic steps are consistent with the ones we explained above.

This article has been included in http://www.flydean.com/14-5-netty-frame-decoder/

The most popular interpretation, the most profound dry goods, the most concise tutorials, and many tricks you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", understand technology, understand you better!