Netty series: built-in Frame detection

[toc]

Introduction

In the last article, we talked about how to customize the encoding and decoder in netty, but the customization is still quite complicated. Generally, when there is no special need, everyone hopes the simpler the better. The difficulty is to find the segmentation in ByteBuf Click to divide the ByteBuf into units that can be processed. Today this article talks about the segmentation processing mechanism that comes with netty.

Frame detection

In the previous chapter, we mentioned that we need a means to distinguish different data in ByteBuf, that is, to find the split point of different data in ByteBuf. If the ByteBuf is first divided into independent ByteBufs, then it will be much simpler to process the independent ByteBufs.

Netty provides four segmentation point encoders, which we can call Frame detection. They are DelimiterBasedFrameDecoder, FixedLengthFrameDecoder, LengthFieldBasedFrameDecoder, and LineBasedFrameDecoder.

These classes are all subclasses of ByteToMessageDecoder, and we will introduce them one by one.

DelimiterBasedFrameDecoder

The first is DelimiterBasedFrameDecoder. Looking at the name, you know that this is a decoder that divides bytebuf according to delimiter. What is delimiter?

There is a Delimiters class in netty, which specifically defines the divided characters. There are mainly two delimiters, namely nulDelimiter and lineDelimiter:

public static ByteBuf[] nulDelimiter() {
        return new ByteBuf[] {
                Unpooled.wrappedBuffer(new byte[] { 0 }) };
    }

    public static ByteBuf[] lineDelimiter() {
        return new ByteBuf[] {
                Unpooled.wrappedBuffer(new byte[] { '\r', '\n' }),
                Unpooled.wrappedBuffer(new byte[] { '\n' }),
        };
    }

nullDelimiter is used to deal with 0x00, mainly used to deal with Flash XML socket or other similar protocols.

lineDelimiter is used to process carriage returns and line feeds, and is mainly used in the processing of text files.

For DelimiterBasedFrameDecoder, if there are multiple delimiters, it will choose the shortest ByteBuf segmentation. For example, if we use DelimiterBasedFrameDecoder(Delimiters.lineDelimiter()), because there are actually two segmentation methods in lineDelimiter, press Enter +Line break or line break, if you encounter the following situations:

   +--------------+
   | ABC\nDEF\r\n |
   +--------------+

DelimiterBasedFrameDecoder will choose the shortest segmentation result, which means that the above content will be divided into:

   +-----+-----+
   | ABC | DEF |
   +-----+-----+

Instead of

   +----------+
   | ABC\nDEF |
   +----------+

FixedLengthFrameDecoder

This class divides ByteBuf into a fixed length, for example, the following 4 bytes of information are received:

   +---+----+------+----+
   | A | BC | DEFG | HI |
   +---+----+------+----+

If a FixedLengthFrameDecoder(3) is used, the ByteBuf above will be divided into the following parts:

   +-----+-----+-----+
   | ABC | DEF | GHI |
   +-----+-----+-----+

LengthFieldBasedFrameDecoder

This class is more flexible, and can take out subsequent byte arrays based on the length field in the data. LengthFieldBasedFrameDecoder is very flexible. It has 4 properties to control them, namely lengthFieldOffset, lengthFieldLength, lengthAdjustment and initialBytesToStrip.

lengthFieldOffset is the starting position of the length field, lengthFieldLength is the length of the length field itself, lengthAdjustment is the adjustment of the target data length, and initialBytesToStrip is the number of bytes that need to be deleted during the decryption process. Can't understand? It doesn't matter, let's give a few examples.

First look at the simplest one:

   lengthFieldOffset   = 0
   lengthFieldLength   = 2
   lengthAdjustment    = 0
   initialBytesToStrip = 0 

   BEFORE DECODE (14 bytes)         AFTER DECODE (14 bytes)
   +--------+----------------+      +--------+----------------+
   | Length | Actual Content |----->| Length | Actual Content |
   | 0x000C | "HELLO, WORLD" |      | 0x000C | "HELLO, WORLD" |
   +--------+----------------+      +--------+----------------+

The above setting means that length starts from bit 0 and the length is 2 bytes. Among them, Ox00C=12, which is also the length of "HELLO, WORLD".

If you don't want the Length field, you can delete length by setting initialBytesToStrip:

   lengthFieldOffset   = 0
   lengthFieldLength   = 2
   lengthAdjustment    = 0
   initialBytesToStrip = 2 (= length 字段的长度)
  
   BEFORE DECODE (14 bytes)         AFTER DECODE (12 bytes)
   +--------+----------------+      +----------------+
   | Length | Actual Content |----->| Actual Content |
   | 0x000C | "HELLO, WORLD" |      | "HELLO, WORLD" |
   +--------+----------------+      +----------------+

lengthAdjustment is to adjust the value of the Length field, because in some cases the Length field may contain the length of the entire data, which is Length+ content, so it needs to be adjusted during parsing. For example, in the following example, the actual length is actually 0x0C , But the incoming is 0x0E, so you need to subtract the length of the Length field by 2, which is to set the lengthAdjustment to -2.

   lengthFieldOffset   =  0
   lengthFieldLength   =  2
   lengthAdjustment    = -2 (= Length字段的长度)
   initialBytesToStrip =  0

   BEFORE DECODE (14 bytes)         AFTER DECODE (14 bytes)
   +--------+----------------+      +--------+----------------+
   | Length | Actual Content |----->| Length | Actual Content |
   | 0x000E | "HELLO, WORLD" |      | 0x000E | "HELLO, WORLD" |
   +--------+----------------+      +--------+----------------+

LineBasedFrameDecoder

LineBasedFrameDecoder specializes in processing the end of a line in a text file. That is, "\n" and "\r\n", it is very similar to DelimiterBasedFrameDecoder, but DelimiterBasedFrameDecoder is more general.

Summarize

With the above 4 frame detection devices, you can add these frame detections in pipline first, and then add a custom handler, so you don't have to consider the length of reading ByteBuf in the custom handler.

For example, in StringDecoder, if LineBasedFrameDecoder has been used, then in the decode method you can assume that the incoming ByteBuf is a line of string, then you can directly use it like this:

    protected void decode(ChannelHandlerContext ctx, ByteBuf msg, List<Object> out) throws Exception {
        out.add(msg.toString(charset));
    }

Is not it simple?

This article has been included in http://www.flydean.com/15-netty-buildin-frame-detection/

The most popular interpretation, the most profound dry goods, the most concise tutorial, and many tips you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", know technology, know you better!