How to solve the problem of unpacking and sticking in Netty through a large number of actual cases?

The TCP transmission protocol is based on data stream transmission, and streaming data has no boundaries. When the client sends data to the server, it may split a complete data message into multiple small messages for transmission. , It is also possible to combine multiple messages into one large message for transmission.

In this case, the situation shown in Figure 3-1 may occur.

• The server happened to read two complete data packets A and B, and there was no unpacking/sticking problem;
• The server receives the data packet that A and B are glued together, and the server needs to parse out A and B;
• The server receives the complete A and B part of the data packet B-1, the server needs to parse out the complete A, and wait for the complete B data packet to be read;
• The server receives a part of A's data packet A-1, and needs to wait for the complete A data packet to be received at this time;
• Data packet A is relatively large, and the server needs to receive data packet A several times.

<center>Figure 3-1 Sticking and unpacking problems</center>

Due to the unpacking/sticking problem, it is difficult for the receiver to define the boundary of the data packet, so incomplete data may be read, causing problems in data analysis.

Actual combat of unpacking and sticking problems

The following demonstrates an unpacking sticking problem

PackageNettyServer

public class PackageNettyServer {

    public static void main(String[] args) {
        EventLoopGroup bossGroup=new NioEventLoopGroup();
        EventLoopGroup workGroup=new NioEventLoopGroup();
        try{
            ServerBootstrap serverBootstrap=new ServerBootstrap();
            serverBootstrap.group(bossGroup,workGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new SimpleServerHandler());
                        }
                    });
            ChannelFuture channelFuture=serverBootstrap.bind(8080).sync(); //绑定端口
            channelFuture.channel().closeFuture().sync();
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            bossGroup.shutdownGracefully();
            workGroup.shutdownGracefully();
        }
    }
}

SimpleServerHandler

public class SimpleServerHandler extends ChannelInboundHandlerAdapter {
    private int count;
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        ByteBuf in=(ByteBuf) msg;
        byte[] buffer=new byte[in.readableBytes()]; //长度为可读的字节数
        in.readBytes(buffer); //读取到字节数组中
        String message=new String (buffer,"UTF-8");
        System.out.println("服务端收到的消息内容："+message+"\n服务端收到的消息数量"+(++count));
        ByteBuf resBB= Unpooled.copiedBuffer(UUID.randomUUID().toString(), Charset.forName("utf-8"));
        ctx.writeAndFlush(resBB);
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        ctx.close();//关闭连接
    }
}

PackageNettyClient

public class PackageNettyClient {

    public static void main(String[] args) {
        EventLoopGroup eventLoopGroup=new NioEventLoopGroup();
        try {
            Bootstrap bootstrap = new Bootstrap();
            bootstrap.group(eventLoopGroup)
                .channel(NioSocketChannel.class)
                .handler(new ChannelInitializer<SocketChannel>() {
                    @Override
                    protected void initChannel(SocketChannel ch) throws Exception {
                        ch.pipeline().addLast(new SimpleClientHandler());
                    }
                });
            ChannelFuture channelFuture=bootstrap.connect("localhost",8080).sync();
            channelFuture.channel().closeFuture().sync();
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            eventLoopGroup.shutdownGracefully();
        }
    }
}

SimpleClientHandler

public class SimpleClientHandler extends ChannelInboundHandlerAdapter {

    private int count;
    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        System.out.println("客户端和服务端成功建立连接");
        //客户端和服务端建立连接后，发送十次消息给服务端
        for (int i = 0; i < 10; i++) {
            ByteBuf buf= Unpooled.copiedBuffer("客户端消息"+i, Charset.forName("utf-8"));
            ctx.writeAndFlush(buf);
        }
        super.channelActive(ctx);
    }

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        //接收服务端发过来的消息
        System.out.println("接收到服务端返回的信息");
        ByteBuf buf=(ByteBuf)msg;
        byte[] buffer=new byte[buf.readableBytes()];
        buf.readBytes(buffer);
        String message=new String(buffer,Charset.forName("utf-8"));
        System.out.println("客户端收到的消息内容为："+message);
        System.out.println("客户端收到的消息数量为："+(++count));
        super.channelRead(ctx, msg);
    }
}

After running the above cases, there will be problems with sticking and unpacking.

Application layer defines communication protocol

How to solve the problem of unpacking and sticking?

Generally, we define the communication protocol at the application layer. In fact, the idea is also very simple, that is, the communication parties agree on a communication message protocol. After the server receives the message, it will decode it according to the agreed protocol to avoid sticking and unpacking problems.

In fact, it’s not difficult for everyone to think about this issue in depth. The reason why the content parsing error on the receiving end of the message after unpacking and sticking the package is because the program cannot recognize a complete message, that is, you don’t know how to unpack it. Combine the messages in one into one complete message, and split the sticky packet data into multiple complete messages according to a certain rule. So thinking from this perspective, we only need to make a recognition rule agreed upon by both parties to the message.

Message length is fixed

Each data message needs a fixed length. When the receiver reads a fixed length message cumulatively, it is considered that a complete message has been obtained. When the sender's data is less than the fixed length, it needs to fill in the space. together.

As shown in Figure 3-2, suppose we fix the message length to 4, then the message that does not reach the length needs to be filled with a space so that the message can form a whole.

<center>Figure 3-2</center>

This method is very simple, but the shortcomings are also obvious. For messages with no fixed length, it is not clear how to set the length, and if the length is set too large, it will cause a waste of bytes. If the length is too small, it will affect the message transmission. Will adopt this approach.

Specific separator

Since there is no way to divide the message by a fixed length, can we add a delimiter to the message? Then the receiver splits the message according to a specific delimiter. For example, we use \r\n to split, as shown in Figure 3-3.

<center>Figure 3-3</center>

In the use scenario of a specific separator, you need to pay attention to the separator and the characters in the message body not to conflict, otherwise there will be a problem of incorrect message splitting.

Message length plus message content plus separator

Data communication is based on the message length + message content + delimiter. You have learned this in Redis before. The message protocol of redis is defined as follows.

*3\r\n$3\r\nSET\r\n$4\r\nname\r\n$3\r\nmic

It can be found that the message message contains three dimensions

• Message length
• Message separator
• Message content

This method is a very common protocol in the project. First, the total length in the message header is used to determine the number of parameters carried by the current complete message. Then in the message body, use the length of the message content and the message body as a combination, and finally divide it by \r\n. After the server receives this message, it can parse it according to the rule to get a complete command for execution.

Message protocol in Zookeeper

The Jute protocol is used in Zookeeper, which is a zookeeper custom message protocol. The request protocol definition is shown in Figure 3-4.

xid is used to record the sequence number of the client request, which is used to ensure the response sequence of a single client request. type represents the type of operation requested. Common ones include creating nodes, deleting nodes, and obtaining node data.
The request body part of the protocol refers to the main content part of the request, which contains all the operation content of the request. The structure of the request body is different for different request types.

<img src="https://mic-blob-bucket.oss-cn-beijing.aliyuncs.com/202111122058016.png" alt="img" style="zoom:80%;" />

<center>Figure 3-4</center>

The definition of the response protocol is shown in Figure 3-5.

The xid in the response header of the protocol is the same as the xid in the request header mentioned above, and the response only returns the original value of xid in the request. zxid represents the latest transaction ID currently on the ZooKeeper server. err is an error code. When an abnormal situation occurs during request processing, it will be identified in this error code. The response body part of the protocol refers to the body content part of the response, which contains all the returned data of the response. The structure of the response body part is different for different response types.

<img src="https://mic-blob-bucket.oss-cn-beijing.aliyuncs.com/202111122058905.png" alt="img" style="zoom: 67%;" />

<center>Figure 3-5</center>

Codecs in Netty

In Netty, some commonly used codecs are provided by default to solve the problem of unpacking and sticking. The following briefly demonstrates the use of several decoders.

FixedLengthFrameDecoder decoder

The principle of the fixed length decoder FixedLengthFrameDecoder is very simple. It is to set a fixed message size frameLength through the construction method. No matter how much data the receiver receives at a time, it will decode in strict accordance with the frameLength.

If the cumulatively read messages with a length of frameLength, the decoder will think that a complete message has been obtained. If the message length is less than frameLength, the decoder will wait for the arrival of subsequent data packets and return after knowing that the specified length is obtained .

The usage method is as follows, add a FixedLengthFrameDecoder on the Server side of the code demonstrated in section 3.3, with a length of 10.

ServerBootstrap serverBootstrap=new ServerBootstrap();
serverBootstrap.group(bossGroup,workGroup)
    .channel(NioServerSocketChannel.class)
    .childHandler(new ChannelInitializer<SocketChannel>() {
        @Override
        protected void initChannel(SocketChannel ch) throws Exception {
            ch.pipeline()
                .addLast(new FixedLengthFrameDecoder(10)) //增加解码器
                .addLast(new SimpleServerHandler());
        }
    });

DelimiterBasedFrameDecoder decoder

Special delimiter decoder: DelimiterBasedFrameDecoder, it has the following properties

• delimiters , delimiters specifies a special delimiter, the parameter type is ByteBuf, ByteBuf can pass an array, which means that we can specify multiple delimiters at the same time, but in the end we will choose the shortest delimiter for splitting.

  For example, the message body received by the receiver is

  hello\nworld\r\n

  At this time, specify multiple delimiters \n and \r\n , then finally the shortest delimiter will be selected for decoding, and the following data will be obtained

  hello | world |

• maxLength indicates the maximum length of the message. If the specified delimiter is not detected after maxLength is exceeded, a TooLongFrameException will be thrown.
• failFast indicates a fault tolerance mechanism, which is used in conjunction with maxLength. If failFast=true, a TooLongFrameException will be thrown immediately after maxLength is exceeded, and no decoding will be performed. If failFast=false, TooLongFrameException will not be thrown until a complete message is decoded

• stripDelimiter , its function is to determine whether the delimiter is removed from the decoded message. If stripDelimiter=false and the specified delimiter is \n , the data decoding method is as follows.

  hello\nworld\r\n

  When stripDelimiter=false, get

  hello\n | world\r\n

DecoderNettyServer

Let's demonstrate the usage of DelimiterBasedFrameDecoder.

public class DecoderNettyServer {

    public static void main(String[] args) {
        EventLoopGroup bossGroup = new NioEventLoopGroup();
        EventLoopGroup workGroup = new NioEventLoopGroup();
        try {
            ServerBootstrap serverBootstrap = new ServerBootstrap();
            serverBootstrap.group(bossGroup, workGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel ch) throws Exception {
                            ByteBuf delimiter= Unpooled.copiedBuffer("&".getBytes());
                            ch.pipeline()
                                    .addLast(new DelimiterBasedFrameDecoder(10,true,true,delimiter))
                                    .addLast(new PrintServerHandler());
                        }
                    });
            ChannelFuture channelFuture = serverBootstrap.bind(8080).sync(); //绑定端口
            channelFuture.channel().closeFuture().sync();
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {

        }
    }
}

PrintServerHandler

Define an ordinary Inbound and print the received data.

public class PrintServerHandler extends ChannelInboundHandlerAdapter {

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        ByteBuf buf=(ByteBuf)msg;
        System.out.println("Receive client Msg:"+buf.toString(CharsetUtil.UTF_8));
    }
}

Demonstration method

• Enter the cmd command window, execute telnet localhost 8080 Enter
• Ctrl+] key combination in the telnet window to enter a telnet interface
• Continue to press Enter on this interface to enter a new window. At this time, you can start to input characters. At this time, the command window will have data write-back.
• Start typing the characters hello&world, you can see the demo effect

LengthFieldBasedFrameDecoder decoder

LengthFieldBasedFrameDecoder is a length domain decoder. It is the most commonly used decoder to solve unpacking and sticking. It can basically cover most scenes based on length unpacking. Among them, the open source message middleware RocketMQ uses this decoder for decoding.

First, let me explain the core parameters of the decoder

• lengthFieldOffset, the offset of the length field, which is the starting position of storing the length data
• lengthFieldLength, the number of bytes occupied by the length field lock
• lengthAdjustment, in some more complex protocol designs, the length field not only contains the length of the message, but also contains other data such as version number, data type, data status, etc. At this time, we can use lengthAdjustment to modify, its value = packet body The length value-the value of the length field
• initialBytesToStrip, the initial number of bytes that need to be skipped after decoding, that is, the starting position of the message content field
• lengthFieldEndOffset, the offset of the end of the length field, the value of this attribute=lengthFieldOffset+lengthFieldLength

The above parameters are more difficult to understand. Let's illustrate through a few cases.

Decoding of message length + message content

Suppose there is a data packet composed of length and message content as shown in Figure 3-6, where length represents the length of the message, expressed in hexadecimal and occupies a total of 2 bytes, then the codec parameters corresponding to the protocol are set as follows .

• lengthFieldOffset=0, because the Length field is at the beginning of the message
• lengthFieldLength=2, the fixed length of the protocol design is 2 bytes
• lengthAdjustment=0, length field warranty letter message length, no need to modify
• initialBytesToStrip=0, the decoded content is Length+content, and there is no need to skip any initial bytes.

<center>Figure 3-6</center>

Truncate the decoded result

If we want the decoded result to contain only the message content, the other parts remain unchanged, as shown in Figure 3-7. The corresponding decoder parameter combination is as follows

• lengthFieldOffset=0, because the Length field is at the beginning of the message
• lengthFieldLength=2, the fixed length of the protocol design
• lengthAdjustment=0, the Length field only contains the length of the message, no correction is required
• initialBytesToStrip=2, skip the byte length of the length field, ByteBuf only contains the Content field after decoding.

<center>Figure 3-7</center>

The length field contains the content of the message

As shown in Figure 3-8, if the Length field contains the length of the Length field itself and the number of bytes occupied by the Content field, then the value of Length is 0x00d (2+11=13 bytes). In this case, decode The parameter combination of the device is as follows

• lengthFieldOffset=0, because the Length field is at the beginning of the message
• lengthFieldLength=2, the fixed length of the protocol design
• lengthAdjustment=-2, the length field is 13 bytes, and 2 needs to be subtracted to be the length required for unpacking.
• initialBytesToStrip=0, the content after decoding is still Length+Content, no need to skip any initial bytes

<center>Figure 3-8</center>

Decoding based on length field offset

As shown in Figure 3-9, the Length field is no longer the starting position of the packet. The value of the Length field is 0x000b, which means that the content field occupies 11 bytes. Then the parameter configuration of the decoder is as follows:

• lengthFieldOffset=2, need to skip the 2 bytes occupied by Header, which is the starting position of Length
• lengthFieldLength=2, the fixed length of the protocol design
• lengthAdjustment=0, the Length field only contains the length of the message, no correction is required
• initialBytesToStrip=0, the content after decoding is still Length+Content, no need to skip any initial bytes

<center>Figure 3-9</center>

Decoding based on length offset and length correction

As shown in Figure 3-10, there are hdr1 and hdr2 fields before and after the Length field, each occupying 1 byte, so the length field needs to be cheaper, and lengthAdjustment needs to be modified. The relevant parameter configuration is as follows.

• lengthFieldOffset=1, need to skip 1 byte occupied by hdr1, is the starting position of Length
• lengthFieldLength=2, the fixed length of the protocol design
• lengthAdjustment=1, because hdr2+content occupies a total of 1+11=12 bytes, so the Length field value (11 bytes) plus lengthAdjustment(1) can get the content of hdr2+Content (12 bytes)
• initialBytesToStrip=3, skip the hdr1 and length fields after decoding, a total of 3 bytes

<center>Figure 3-10</center>

Decoder combat

For example, we define the following message header, the client sends data through the message protocol, and the server needs to decode the message after receiving the message

First define the client, where the Length part can be realized by using the LengthFieldPrepender that Netty comes with. It can calculate the binary byte length of the currently sent message, and then add the length to the buffer header of ByteBuf.

public class LengthFieldBasedFrameDecoderClient {

    public static void main(String[] args) {
        EventLoopGroup workGroup=new NioEventLoopGroup();
        Bootstrap b=new Bootstrap();
        b.group(workGroup)
            .channel(NioSocketChannel.class)
            .handler(new ChannelInitializer<SocketChannel>() {
                @Override
                protected void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline()
                        //如果协议中的第一个字段为长度字段，
                        // netty提供了LengthFieldPrepender编码器，
                        // 它可以计算当前待发送消息的二进制字节长度，将该长度添加到ByteBuf的缓冲区头中
                        .addLast(new LengthFieldPrepender(2,0,false))
                        //使用StringEncoder，在通过writeAndFlush时，不需要自己转化成ByteBuf
                        //StringEncoder会自动做这个事情
                        .addLast(new StringEncoder())
                        .addLast(new ChannelInboundHandlerAdapter(){
                            @Override
                            public void channelActive(ChannelHandlerContext ctx) throws Exception {
                                ctx.writeAndFlush("i am request!");
                                ctx.writeAndFlush("i am a another request!");
                            }
                        });
                }
            });
        try {
            ChannelFuture channelFuture=b.connect("localhost",8080).sync();
            channelFuture.channel().closeFuture().sync();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }finally {
            workGroup.shutdownGracefully();
        }
    }
}

When the above code runs, you will get two messages.

The following is the code on the Server side, which adds the LengthFieldBasedFrameDecoder decoder. The values of two parameters are as follows

• lengthFieldLength: 2, which means that the number of bytes occupied by length is 2
• initialBytesToStrip: 2, which means that after decoding, skip 2 bytes of length and get the content

public class LengthFieldBasedFrameDecoderServer {

    public static void main(String[] args) {
        EventLoopGroup bossGroup=new NioEventLoopGroup();
        EventLoopGroup workGroup=new NioEventLoopGroup();
        try{
            ServerBootstrap serverBootstrap=new ServerBootstrap();
            serverBootstrap.group(bossGroup,workGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel ch) throws Exception {
                            ch.pipeline()
                                    .addLast(new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE,0,2,0,2))
                                    .addLast(new StringDecoder())
                                    .addLast(new ChannelInboundHandlerAdapter(){
                                        @Override
                                        public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
                                            System.out.println("receive message:"+msg);
                                        }
                                    });
                        }
                    });
            ChannelFuture channelFuture=serverBootstrap.bind(8080).sync(); //绑定端口
            channelFuture.channel().closeFuture().sync();
        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            bossGroup.shutdownGracefully();
            workGroup.shutdownGracefully();
        }
    }
}

Summarize

The several commonly used decoders we analyzed above only helped us solve the problem of half-packet and sticky packet, and finally let the receiver receive a complete and effective request message and encapsulate it in ByteBuf, and whether the message content has Other encoding methods, such as serialization, need to be parsed separately.

In addition, many middlewares will define their own message protocols. In addition to solving the problem of sticky packets and half packets, these message protocols will also transmit some other meaningful data, such as zookeeper's jute, dubbo framework's dubbo protocol, etc. .

Copyright statement: All articles in this blog, except for special statements, adopt the CC BY-NC-SA 4.0 license agreement. Please indicate the reprint from Mic takes you to learn architecture!
If this article is helpful to you, please help me to follow and like. Your persistence is the motivation for my continuous creation. Welcome to follow the WeChat public account of the same name for more technical dry goods!