Netty series: detailed explanation of ByteBuf in netty

Introduction

The class used for information carrying and communication in netty is called ByteBuf. It can be seen from the name that this is the buffer area of Byte. So what are the characteristics of ByteBuf? Lets come look.

ByteBuf detailed

netty provides an io.netty.buffer package, which defines various types of ByteBuf and its derived types.

The basis of netty Buffer is the ByteBuf class, which is an abstract class. Other Buffer classes are basically derived from this class. This class also defines the tone of netty's overall Buffer.

Let's first look at the definition of ByteBuf:

public abstract class ByteBuf implements ReferenceCounted, Comparable<ByteBuf> {

ByteBuf implements two interfaces, ReferenceCounted and Comparable. Comparable is an interface that comes with the JDK, which means that the classes can be compared. The ReferenceCounted represents the reference statistics of the object. When a ReferenceCounted is instantiated, its reference count=1, every time the retain() method is called, the count is increased, and the release() method is called to decrease the count. When the count is reduced to 0, the object will be released. If you try to access the released object, an access exception will be reported.

If an object implements ReferenceCounted, and other objects contained in this object also implement ReferenceCounted, then when the count=0 of the container object, other objects inside it will also be released by calling the release() method.

In summary, ByteBuf is a comparable object that can count the number of references. He provides a sequence or random byte access mechanism.

Note that although there is a built-in ByteBuffer class in the JDK, the ByteBuf in netty is a re-implementation of Byte Buffer. They are not related.

Create a buff

ByteBuf is an abstract class and cannot be used directly for instantiation. Although you can use subclasses of ByteBuf for instantiation, netty does not recommend it. netty recommends using io.netty.buffer.Unpooled to create Buff. Unpooled is a tool class that can allocate space, copy or encapsulate operations for ByteBuf.

The following are examples of creating several different ByteBufs:

   import static io.netty.buffer.Unpooled.*;
  
   ByteBuf heapBuffer    = buffer(128);
   ByteBuf directBuffer  = directBuffer(256);
   ByteBuf wrappedBuffer = wrappedBuffer(new byte[128], new byte[256]);
   ByteBuf copiedBuffer  = copiedBuffer(ByteBuffer.allocate(128));

Above we have seen 4 different buff construction methods, ordinary buff, directBuffer, wrappedBuffer and copiedBuffer.

Ordinary buff is a fixed-size heap buff, and directBuffer is a fixed-size direct buff. Direct buff uses off-heap memory, eliminating the need to copy data to the kernel, so the efficiency is higher than ordinary buff.

WrappedBuffer is an encapsulation of existing byte arrays or byte buffers. It can be regarded as a view. When the underlying data changes, the data in the wrapped buffer will also change.

Copied buffer is a deep copy of existing byte arrays, byte buffers or strings, so it is different from wrappedBuffer, and Copied buffer and original data do not share data.

Random access to Buff

Friends who are familiar with collections should know that if you want to randomly access a certain collection, you must access it through index. The same is true for ByteBuf. You can get its capacity through capacity or get its capacity, and then randomly access the bytes in it through the getByte method, as shown below:

        //随机访问
        ByteBuf buffer = heapBuffer;
        for (int i = 0; i < buffer.capacity(); i ++) {
            byte b = buffer.getByte(i);
            System.out.println((char) b);
        }

Sequence read and write

Reading and writing is a bit more complicated than accessing. ByteBuf provides two indexes to locate the position of reading and writing, namely readerIndex and writerIndex. The two indexes control the position of reading and writing respectively.

The following figure shows a buffer is divided into three parts, which are discardable bytes, readable bytes, and writable bytes.

    +-------------------+------------------+------------------+
    | discardable bytes |  readable bytes  |  writable bytes  |
    |                   |     (CONTENT)    |                  |
    +-------------------+------------------+------------------+
    |                   |                  |                  |
    0      <=      readerIndex   <=   writerIndex    <=    capacity

The figure above also shows the relationship between readerIndex, writerIndex and capacity.

Among them, readable bytes are the real content, which can or skip . When these methods are called, the readerIndex will increase synchronously. If it exceeds the range of readable bytes, an IndexOutOfBoundsException will be thrown. By default, readerIndex=0.

The following is an example of traversing readable bytes:

        //遍历readable bytes
        while (directBuffer.isReadable()) {
            System.out.println(directBuffer.readByte());
        }

First determine whether to call the readByte method by judging whether it is readable.

Writable bytes is an undetermined area, waiting to be filled. It can be operated by calling the write* method, and the writerIndex will be updated synchronously. Similarly, if the space is not enough, IndexOutOfBoundsException will also be thrown. By default, the newly allocated writerIndex = 0, and the writerIndex of the wrapped or copied buffer = the capacity of buf.

The following is an example of using writable Byte:

        //写入writable bytes
        while (wrappedBuffer.maxWritableBytes() >= 4) {
            wrappedBuffer.writeInt(new Random().nextInt());
        }

Discardable bytes are bytes that have been read. Initially, its value is =0. Whenever the readerIndex moves to the right, the space of Discardable bytes will increase. If you want to completely delete or reset the Discardable bytes, you can call the discardReadBytes() method, which will delete the Discardable bytes space and put the extra space in the writable bytes, as shown below:

调用 discardReadBytes() 之前：

    +-------------------+------------------+------------------+
    | discardable bytes |  readable bytes  |  writable bytes  |
    +-------------------+------------------+------------------+
    |                   |                  |                  |
    0      <=      readerIndex   <=   writerIndex    <=    capacity


调用 discardReadBytes()之后：

    +------------------+--------------------------------------+
    |  readable bytes  |    writable bytes (got more space)   |
    +------------------+--------------------------------------+
    |                  |                                      |

readerIndex (0) <= writerIndex (decreased) <= capacity

Note that although the number of writable bytes has increased, its content is uncontrollable, and there is no guarantee that the content inside is empty or unchanged.

Calling the clear() method will clear readerIndex and writerIndex. Note that the clear method will only set the values of readerIndex and writerIndex, but will not clear content. See the diagram below:

调用 clear()之前：

    +-------------------+------------------+------------------+
    | discardable bytes |  readable bytes  |  writable bytes  |
    +-------------------+------------------+------------------+
    |                   |                  |                  |
    0      <=      readerIndex   <=   writerIndex    <=    capacity


调用 clear()之后：

    +---------------------------------------------------------+
    |             writable bytes (got more space)             |
    +---------------------------------------------------------+
    |                                                         |
    0 = readerIndex = writerIndex            <=            capacity

search

ByteBuf provides a single byte search function, such as indexOf(int, int, byte) and bytesBefore(int, int, byte) two methods.

If you want to search for ByteBuf traversal, you can use forEachByte(int, int, ByteProcessor), this method receives a ByteProcessor for complex processing.

Other derived buffer methods

ByteBuf also provides many methods to create derived buffers, as shown below:

duplicate()
slice()
slice(int, int)
readSlice(int)
retainedDuplicate()
retainedSlice()
retainedSlice(int, int)
readRetainedSlice(int)

It should be noted that these bufs are derivatives based on the existing bufs, and their underlying content is the same, only the readerIndex, writerIndex and the marked index are different. So they share data with the original buf. If you want to create a new buffer, you can use the copy() method or the aforementioned Unpooled.copiedBuffer.

In the previous section, we mentioned that ByteBuf is a ReferenceCounted, and this feature is used in the derivative buf. We know that when the retain() method is called, the reference count will increase, but for the duplicate(), slice(), slice(int, int) and readSlice(int) methods, although they are also references, retain is not called () method, so that the original data will be recovered after any Buf calls the release() method.

If you don't want the above side effects, you can replace the methods with retainedDuplicate(), retainedSlice(), retainedSlice(int, int) and readRetainedSlice(int). These methods will call the retained() method to add a reference.

And existing JDK type conversion

As mentioned earlier, ByteBuf is a rewrite of ByteBuffer, and they are different implementations. Although these two are different, it does not prevent ByteBuf to ByteBuffer.

Of course, the simplest conversion is to convert ByteBuf into a byte array byte[]. If you want to convert to a byte array, you can call hasArray() first to judge, and then call array() method to convert.

The same ByteBuf can also be converted into ByteBuffer. You can call nioBufferCount() first to determine the number of ByteBuffers that can be converted into ByteBuffers, and then call nioBuffer() to convert.

The returned ByteBuffer is a sharing or copying of the existing buf, and the modification of the position and limit of the buffer after the return will not affect the original buf.

Finally, use the toString(Charset) method to convert ByteBuf to String.

Summarize

ByteBuf is the underlying foundation of netty, and is the bearer of data transmission. A deep understanding of ByteBuf can understand the design idea of netty, which is very good.

For the examples in this article, please refer to: learn-netty4

This article has been included in http://www.flydean.com/02-netty-bytebuf/

The most popular interpretation, the most profound dry goods, the most concise tutorial, and many tips you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", know technology, know you better!