Netty series: detailed explanation of kequeue transmission protocol

Introduction

In the previous chapters, we introduced that kequeue or epoll can be used in netty to achieve a more efficient native transmission method. So what is the difference between kequeue and epoll and NIO transport protocol?

This chapter will take kequeue as an example for in-depth discussion.

In the native example we introduced above, there are several classes about kqueue, namely KQueueEventLoopGroup, KQueueServerSocketChannel and KQueueSocketChannel. By simply replacing and adding the corresponding dependency package, we can easily replace the ordinary NIO netty service with native Kqueue service.

Time to unravel the secrets of Kqueue.

KQueueEventLoopGroup

eventLoop and eventLoopGroup are used to accept events and event processing. Let's first look at the definition of KQueueEventLoopGroup:

 public final class KQueueEventLoopGroup extends MultithreadEventLoopGroup

As a MultithreadEventLoopGroup, a newChild method must be implemented to create a child EventLoop. In KQueueEventLoopGroup, in addition to the constructor, the additional method that needs to be implemented is newChild:

 protected EventLoop newChild(Executor executor, Object... args) throws Exception {
        Integer maxEvents = (Integer) args[0];
        SelectStrategyFactory selectStrategyFactory = (SelectStrategyFactory) args[1];
        RejectedExecutionHandler rejectedExecutionHandler = (RejectedExecutionHandler) args[2];
        EventLoopTaskQueueFactory taskQueueFactory = null;
        EventLoopTaskQueueFactory tailTaskQueueFactory = null;

        int argsLength = args.length;
        if (argsLength > 3) {
            taskQueueFactory = (EventLoopTaskQueueFactory) args[3];
        }
        if (argsLength > 4) {
            tailTaskQueueFactory = (EventLoopTaskQueueFactory) args[4];
        }
        return new KQueueEventLoop(this, executor, maxEvents,
                selectStrategyFactory.newSelectStrategy(),
                rejectedExecutionHandler, taskQueueFactory, tailTaskQueueFactory);
    }

All parameters in newChild are passed in from the constructor of KQueueEventLoopGroup. In addition to maxEvents, selectStrategyFactory and rejectedExecutionHandler, it can also receive two parameters, taskQueueFactory and tailTaskQueueFactory, and finally pass these parameters to the constructor of KQueueEventLoop, and finally return a KQueueEventLoop object.

In addition, before using KQueueEventLoopGroup, we also need to ensure that Kqueue is available in the system. This judgment is achieved by calling KQueue.ensureAvailability(); .

KQueue.ensureAvailability first determines whether the system property io.netty.transport.noNative is defined. If it is set, it means that native transport is disabled, and there is no need to make further judgments.

If io.netty.transport.noNative is not defined, then Native.newKQueue() will be called to try to obtain a kqueue's FileDescriptor from native. If there is no exception in the above acquisition process, it means that kqueue is in the native method exists, we can continue to use it.

The following is the code to determine whether kqueue is available:

 static {
        Throwable cause = null;
        if (SystemPropertyUtil.getBoolean("io.netty.transport.noNative", false)) {
            cause = new UnsupportedOperationException(
                    "Native transport was explicit disabled with -Dio.netty.transport.noNative=true");
        } else {
            FileDescriptor kqueueFd = null;
            try {
                kqueueFd = Native.newKQueue();
            } catch (Throwable t) {
                cause = t;
            } finally {
                if (kqueueFd != null) {
                    try {
                        kqueueFd.close();
                    } catch (Exception ignore) {
                        // ignore
                    }
                }
            }
        }
        UNAVAILABILITY_CAUSE = cause;
    }

KQueueEventLoop

KQueueEventLoop is created from KQueueEventLoopGroup to perform specific IO tasks.

Let's first look at the definition of KQueueEventLoop:

 final class KQueueEventLoop extends SingleThreadEventLoop

Whether it is NIO or KQueue or Epoll, because they use more advanced IO technology, the EventLoop they use is SingleThreadEventLoop, which means that a single thread is enough.

Like KQueueEventLoopGroup, KQueueEventLoop also needs to determine whether the current system environment supports kqueue:

 static {
        KQueue.ensureAvailability();
    }

As mentioned in the previous section, KQueueEventLoopGroup will call the constructor of KQueueEventLoop to return an eventLoop object. Let's first look at the constructor of KQueueEventLoop:

 KQueueEventLoop(EventLoopGroup parent, Executor executor, int maxEvents,
                    SelectStrategy strategy, RejectedExecutionHandler rejectedExecutionHandler,
                    EventLoopTaskQueueFactory taskQueueFactory, EventLoopTaskQueueFactory tailTaskQueueFactory) {
        super(parent, executor, false, newTaskQueue(taskQueueFactory), newTaskQueue(tailTaskQueueFactory),
                rejectedExecutionHandler);
        this.selectStrategy = ObjectUtil.checkNotNull(strategy, "strategy");
        this.kqueueFd = Native.newKQueue();
        if (maxEvents == 0) {
            allowGrowing = true;
            maxEvents = 4096;
        } else {
            allowGrowing = false;
        }
        this.changeList = new KQueueEventArray(maxEvents);
        this.eventList = new KQueueEventArray(maxEvents);
        int result = Native.keventAddUserEvent(kqueueFd.intValue(), KQUEUE_WAKE_UP_IDENT);
        if (result < 0) {
            cleanup();
            throw new IllegalStateException("kevent failed to add user event with errno: " + (-result));
        }
    }

The incoming maxEvents represents the maximum number of events that this KQueueEventLoop can accept. If maxEvents=0, it means that the event capacity of KQueueEventLoop can be dynamically expanded, and the maximum value is 4096. Otherwise, the event capacity of KQueueEventLoop cannot be expanded.

maxEvents is the size of the array used to construct the changeList and eventList.

A map called channels is also defined in KQueueEventLoop, which is used to save the registered channels:

 private final IntObjectMap<AbstractKQueueChannel> channels = new IntObjectHashMap<AbstractKQueueChannel>(4096);

Take a look at the add and remote methods of the channel:

 void add(AbstractKQueueChannel ch) {
        assert inEventLoop();
        AbstractKQueueChannel old = channels.put(ch.fd().intValue(), ch);
        assert old == null || !old.isOpen();
    }

    void remove(AbstractKQueueChannel ch) throws Exception {
        assert inEventLoop();
        int fd = ch.fd().intValue();
        AbstractKQueueChannel old = channels.remove(fd);
        if (old != null && old != ch) {
            channels.put(fd, old);
            assert !ch.isOpen();
        } else if (ch.isOpen()) {
            ch.unregisterFilters();
        }
    }

You can see that the addition and deletion are AbstractKQueueChannel. We will explain KQueueChannel in detail in the following chapters. Here we only need to know that the key in the channel map is the int value of the FileDescriptor unique to kequeue.

Let's take a look at the most important run method in EventLoop:

 protected void run() {
        for (;;) {
            try {
                int strategy = selectStrategy.calculateStrategy(selectNowSupplier, hasTasks());
                switch (strategy) {
                    case SelectStrategy.CONTINUE:
                        continue;

                    case SelectStrategy.BUSY_WAIT:
          
                    case SelectStrategy.SELECT:
                        strategy = kqueueWait(WAKEN_UP_UPDATER.getAndSet(this, 0) == 1);
                        if (wakenUp == 1) {
                            wakeup();
                        }
                    default:
                }

                final int ioRatio = this.ioRatio;
                if (ioRatio == 100) {
                    try {
                        if (strategy > 0) {
                            processReady(strategy);
                        }
                    } finally {
                        runAllTasks();
                    }
                } else {
                    final long ioStartTime = System.nanoTime();

                    try {
                        if (strategy > 0) {
                            processReady(strategy);
                        }
                    } finally {
                        final long ioTime = System.nanoTime() - ioStartTime;
                        runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                    }

Its logic is to first use selectStrategy.calculateStrategy to obtain the current select strategy, then judge whether the processReady method needs to be executed according to the value of strategy, and finally execute runAllTasks to get the tasks to be executed from the task queue to execute.

selectStrategy.calculateStrategy is used to judge the current select state. By default, there are three states: SELECT, CONTINUE, BUSY_WAIT. All three states are negative:

 int SELECT = -1;

    int CONTINUE = -2;

    int BUSY_WAIT = -3;

Respectively indicate that the current IO is in the block state of slect, or the state of skipping the current IO, and the state of IO loop pull. BUSY_WAIT is a non-blocking IO PULL that kqueue does not support, so it will fallback to SELECT.

In addition to these three states, calculateStrategy returns a positive value indicating the number of tasks currently to be executed.

In the run method, if the result of the strategy is SELECT, then the Native.keventWait method will eventually be called to return the current number of ready events, and the ready events will be placed in the eventList of KQueueEventArray.

If the number of ready events is greater than zero, the processReady method will be called to perform status callback processing on these events.

How to deal with it? The following is the core logic of the processing:

 AbstractKQueueChannel channel = channels.get(fd);

            AbstractKQueueUnsafe unsafe = (AbstractKQueueUnsafe) channel.unsafe();

            if (filter == Native.EVFILT_WRITE) {
                unsafe.writeReady();
            } else if (filter == Native.EVFILT_READ) {
                unsafe.readReady(eventList.data(i));
            } else if (filter == Native.EVFILT_SOCK && (eventList.fflags(i) & Native.NOTE_RDHUP) != 0) {
                unsafe.readEOF();
            }

The fd here is read from eventList:

 final int fd = eventList.fd(i);

According to the fd of eventList, we can get the corresponding KQueueChannel from channels, and then determine the specific operation of KQueueChannel according to the filter state of the event, which is writeReady, readReady or readEOF.

The last is to execute the runAllTasks method. The logic of runAllTasks is very simple, that is, to read tasks from taskQueue and execute them.

KQueueServerSocketChannel and KQueueSocketChannel

KQueueServerSocketChannel is a channel used on the server side:

 public final class KQueueServerSocketChannel extends AbstractKQueueServerChannel implements ServerSocketChannel {

KQueueServerSocketChannel inherits from AbstractKQueueServerChannel. In addition to the constructor, the most important method is newChildChannel:

 @Override
    protected Channel newChildChannel(int fd, byte[] address, int offset, int len) throws Exception {
        return new KQueueSocketChannel(this, new BsdSocket(fd), address(address, offset, len));
    }

This method is used to create a new child channel. From the above code, we can see that the generated child channel is an instance of KQueueSocketChannel.

Its constructor accepts three parameters, which are parent channel, BsdSocket and InetSocketAddress.

 KQueueSocketChannel(Channel parent, BsdSocket fd, InetSocketAddress remoteAddress) {
        super(parent, fd, remoteAddress);
        config = new KQueueSocketChannelConfig(this);
    }

Here fd is the result of socket accept acceptedAddress:

 int acceptFd = socket.accept(acceptedAddress);

Here is the definition of KQueueSocketChannel:

 public final class KQueueSocketChannel extends AbstractKQueueStreamChannel implements SocketChannel {

The relationship between KQueueSocketChannel and KQueueServerSocketChannel is a parent-child relationship. There is a parent method in KQueueSocketChannel to return the ServerSocketChannel object, which is also the serverChannel passed into the KQueueSocketChannel constructor in the newChildChannel method mentioned earlier:

 public ServerSocketChannel parent() {
        return (ServerSocketChannel) super.parent();
    }

Another feature of KQueueSocketChannel is that it supports tcp fastopen. Its essence is to call the connectx method of BsdSocket to transfer data while establishing a connection:

 int bytesSent = socket.connectx(
                                (InetSocketAddress) localAddress, (InetSocketAddress) remoteAddress, iov, true);

Summarize

The above is the detailed introduction of KqueueEventLoop and KqueueSocketChannel, which is basically not much different from NIO, but the performance is excellent.

For more information, please refer to http://www.flydean.com/53-1-netty-kqueue-transport/

The most popular interpretation, the most profound dry goods, the most concise tutorials, and many tricks you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", understand technology, understand you better!