Netty series: the lazy codec in netty

Introduction

The reason why netty is powerful is that it has many very useful codecs built-in. By using these codecs, you can easily build very powerful applications. Today I will tell you about the most basic built-in codecs in netty. .

Built-in encoder in netty

When introducing the netty package, we can see that netty has a lot of artifactIds starting with netty-codec, statistics, there are so many:

netty-codec
netty-codec-http
netty-codec-http2
netty-codec-memcache
netty-codec-redis
netty-codec-socks
netty-codec-stomp
netty-codec-mqtt
netty-codec-haproxy
netty-codec-dns

There are a total of 10 codec packages, of which netty-codec is the most basic one, and the other 9 are extensions and adaptations to different protocol packages. You can see that netty supports common and popular protocol formats, which is very powerful. Because there are so many codecs, it is not easy to explain them. This article will take netty-codec as an example to explain the most basic and most common codec.

Problems to pay attention to when using codec

Although netty provides a very convenient codec codec, as we mentioned in the previous article, some codecs need to be used in conjunction with Frame detection. First use Frame detection to split ByteBuf into one representing the real The ByteBuf of the data is then processed by the built-in codec of netty or a custom codec, so as to achieve the desired effect.

The basic codec built in netty

The basic codec in netty includes base64, bytes, compression, json, marshalling, protobuf, serialization, string and xml.

The following will explain one by one.

base64

This codec is responsible for the conversion between ByteBuf and ByteBuf after base64. Although they are all from ByteBuf to ByteBuf, the content has changed.

There are two key classes, Base64Encoder and Base64Decoder. Because Base64Decoder is a MessageToMessageDecoder, you need to use a DelimiterBasedFrameDecoder to process in advance. Common examples are as follows:

   ChannelPipeline pipeline = ...;
  
   // Decoders
   pipeline.addLast("frameDecoder", new DelimiterBasedFrameDecoder(80, Delimiters.nulDelimiter()));
   pipeline.addLast("base64Decoder", new Base64Decoder());
  
   // Encoder
   pipeline.addLast("base64Encoder", new Base64Encoder());

bytes

Bytes is to convert between the bytes array and ByteBuf. Similarly, before decoding, you also need to use FrameDecoder. The usual way to use it is as follows:

   ChannelPipeline pipeline = ...;
  
   // Decoders
   pipeline.addLast("frameDecoder",
                    new LengthFieldBasedFrameDecoder(1048576, 0, 4, 0, 4));
   pipeline.addLast("bytesDecoder",
                    new ByteArrayDecoder());
  
   // Encoder
   pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
   pipeline.addLast("bytesEncoder", new ByteArrayEncoder());
   

compression

The content of the compression package is relatively rich, mainly for data compression and decompression services. The supported algorithms are as follows:

brotli
Bzip2
FastLZ
JdkZlib
Lz4
Lzf
Snappy
Zlib
Zstandard

Compression is particularly helpful for the transmission of large amounts of data. Through compression, the amount of data to be transmitted can be saved, thereby increasing the transmission speed.

However, compression is calculated using a specific algorithm, so it is a high CPU operation. When we use it, we need to take into account network speed and CPU performance, and get a balance from it.

json

There is only one JsonObjectDecoder class in the json package, which is mainly responsible for converting JSON objects or arrays of Byte streams into JSON objects and arrays.

JsonObjectDecoder is directly a subclass of ByteToMessageDecoder, so it does not need FrameDecoder. It judges the starting position of the Byte array based on the matching of parentheses, so as to distinguish which Byte data belongs to the same Json object or array.

If we want to use JSON to transmit data, this class is very useful.

marshalling

The full name of Marshalling is JBoss Marshalling, which is a method of object serialization produced by JBoss, but the latest API of JBoss Marshalling is still on 2011-04-27. It has not been updated for 10 years. Has it been deprecated?

Therefore, we will not introduce the content of this serialization in detail here, and interested friends can explore it by themselves.

protobuf

Everyone should be familiar with protobuf. It is an information exchange format produced by Google, which can be regarded as a way of serialization. It is a language-neutral, platform-neutral, and extensible structured data serialization mechanism, similar to XML, but smaller, faster, and simpler than XML.

Netty's support for protobuf is that it can convert the message and MessageLite objects in protobuf with ByteBuf.

The two encoders of protobuf are also the direct conversion of message to message, so frame detection is also required. Of course, you can also use other frame detection such as LengthFieldPrepender and LengthFieldBasedFrameDecoder as shown below:

   ChannelPipeline pipeline = ...;
  
   // Decoders
   pipeline.addLast("frameDecoder",
                    new LengthFieldBasedFrameDecoder(1048576, 0, 4, 0, 4));
   pipeline.addLast("protobufDecoder",
                    new ProtobufDecoder(MyMessage.getDefaultInstance()));
  
   // Encoder
   pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
   pipeline.addLast("protobufEncoder", new ProtobufEncoder());

Among them, LengthFieldPrepender will automatically add a length field in front of the field:

之前：
   +----------------+
   | "HELLO, WORLD" |
   +----------------+

之后：
   +--------+----------------+
   + 0x000C | "HELLO, WORLD" |
   +--------+----------------+

Of course, netty has prepared two special frame detections for protobuf, they are ProtobufVarint32FrameDecoder and ProtobufVarint32LengthFieldPrepender. Before explaining these two classes, we need to understand the Base 128 Varints in protobuf.

What is Varints? That is, when serializing integers, the size of the space occupied is different. Small integers take up less space, and large integers take up more space. In this way, there is no need to fix a specific length, which can reduce the length of the data, but it will bring The complexity of the analysis.

So how do you know how many bytes this data needs? In protobuf, the highest bit of each byte is a judgment bit. If this bit is set to 1, it means that the next byte and the byte are together, which means the same number. If this bit is set to 0, it means The next byte has nothing to do with this byte, and the data will end when it reaches this byte.

For example, a byte is 8 bits. If it represents the integer 1, then it can be represented by the following byte:

0000 0001

If a byte cannot hold an integer, then multiple bytes need to be used for connection operations. For example, the following data represents 300:

1010 1100 0000 0010

Why is it 300? First look at the first byte. Its first bit is 1, which means there is another byte behind. Look at the second byte, its first bit is 0, which means it's over. We remove the judgment bit and turn it into the following number:

010 1100 000 0010

At this time, the value of the data cannot be calculated, because in protobuf, the number of bits of byte is reversed, so we need to swap the two bytes above:

000 0010 010 1100 

That is:

10 010 1100 

=256 + 32 + 8 + 4 = 300

In protobuf, Varint is generally used as the length of the field, so netty provides ProtobufVarint32LengthFieldPrepender and ProtobufVarint32FrameDecoder to convert ByteBuf.

For example, add varint length to ByteBuf:

   BEFORE ENCODE (300 bytes)       AFTER ENCODE (302 bytes)
   +---------------+               +--------+---------------+
   | Protobuf Data |-------------->| Length | Protobuf Data |
   |  (300 bytes)  |               | 0xAC02 |  (300 bytes)  |
   +---------------+               +--------+---------------+

Delete the length field of varint when decoding:

   BEFORE DECODE (302 bytes)       AFTER DECODE (300 bytes)
   +--------+---------------+      +---------------+
   | Length | Protobuf Data |----->| Protobuf Data |
   | 0xAC02 |  (300 bytes)  |      |  (300 bytes)  |
   +--------+---------------+      +---------------+

serialization

Serialization is to convert objects into binary data. In fact, all codecs can be serialized. They provide conversion methods between objects and bytes.

Netty also provides two object conversion methods: ObjectDecoder and ObjectEncoder.

It should be noted that these two objects are incompatible with the ObjectInputStream and ObjectOutputStream that come with the JDK. If you want to be compatible, you can use CompactObjectInputStream, CompactObjectOutputStream, and CompatibleObjectEncoder.

string

String is the most commonly used object. Netty provides StringDecoder and StringEncoder for string.

Similarly, before using these two classes, you need to convert the message, usually using LineBasedFrameDecoder to convert by line:

   ChannelPipeline pipeline = ...;
  
   // Decoders
   pipeline.addLast("frameDecoder", new LineBasedFrameDecoder(80));
   pipeline.addLast("stringDecoder", new StringDecoder(CharsetUtil.UTF_8));
  
   // Encoder
   pipeline.addLast("stringEncoder", new StringEncoder(CharsetUtil.UTF_8));

xml

XML is also a very commonly used format, but its volume will be relatively large, and it should be used less now. Netty provides an XmlFrameDecoder for parsing.

Because xml has its own start and end characters, there is no need to do frame detection, just convert directly, such as:

   +-----+-----+-----------+
   | <an | Xml | Element/> |
   +-----+-----+-----------+
转换成：
   +-----------------+
   | <anXmlElement/> |
   +-----------------+

   +-----+-----+-----------+-----+----------------------------------+
   | <an | Xml | Element/> | <ro | ot><child>content</child></root> |
   +-----+-----+-----------+-----+----------------------------------+
   转换成：
   +-----------------+-------------------------------------+
   | <anXmlElement/> | <root><child>content</child></root> |
   +-----------------+-------------------------------------+

It's all possible.

Summarize

Netty provides a lot of excellent codecs to adapt to various application protocols. You can use them to find the differences between different protocols.

This article has been included in http://www.flydean.com/16-netty-buildin-codec-common/

The most popular interpretation, the most profound dry goods, the most concise tutorial, and many tips you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", know technology, know you better!