Come and experience the fast channel, detailed explanation of epoll transmission protocol in netty

Introduction

In the previous chapters, we explained the use and principle of kqueue, then let's look at the use of epoll. Both are more advanced IO methods, and both need to be implemented by native methods. The difference is that Kqueue is used in the mac system, and epoll is used in the liunx system.

Detailed use of epoll

The use of epoll is also very simple. We still use the commonly used chat room as an example to explain the use of epoll.

For the server side, you need to create bossGroup and workerGroup. In NIO, these two groups are NIOEventLoopGroup, and in epoll, you need to use EpollEventLoopGroup:

 EventLoopGroup bossGroup = new EpollEventLoopGroup(1);
        EventLoopGroup workerGroup = new EpollEventLoopGroup();

Then you need to pass bossGroup and workerGroup into ServerBootstrap:

 ServerBootstrap b = new ServerBootstrap();
            b.group(bossGroup, workerGroup)
             .channel(EpollServerSocketChannel.class)
             .handler(new LoggingHandler(LogLevel.INFO))
             .childHandler(new NativeChatServerInitializer());

Note that the channel passed in here is EpollServerSocketChannel, which is specially used to process epoll requests. Other parts are the same as normal NIO services.

Next, let's take a look at the client of epoll. For the client, you need to create an EventLoopGroup. Here we use EpollEventLoopGroup:

 EventLoopGroup group = new EpollEventLoopGroup();

Then pass this group into Bootstrap:

 Bootstrap b = new Bootstrap();
            b.group(group)
             .channel(EpollSocketChannel.class)
             .handler(new NativeChatClientInitializer());

The channel used here is EpollSocketChannel, which is the client's channel corresponding to EpollServerSocketChannel.

EpollEventLoopGroup

First look at the definition of EpollEventLoopGroup:

 public final class EpollEventLoopGroup extends MultithreadEventLoopGroup

Like KqueueEventLoopGroup, EpollEventLoopGroup also inherits from MultithreadEventLoopGroup, which means that it can open multiple threads.

Before using EpollEventLoopGroup, you need to ensure that the epoll-related JNI interfaces have been prepared:

 Epoll.ensureAvailability();

The newChild method is used to generate the child EventLoop of the EpollEventLoopGroup:

 protected EventLoop newChild(Executor executor, Object... args) throws Exception {
        Integer maxEvents = (Integer) args[0];
        SelectStrategyFactory selectStrategyFactory = (SelectStrategyFactory) args[1];
        RejectedExecutionHandler rejectedExecutionHandler = (RejectedExecutionHandler) args[2];
        EventLoopTaskQueueFactory taskQueueFactory = null;
        EventLoopTaskQueueFactory tailTaskQueueFactory = null;

        int argsLength = args.length;
        if (argsLength > 3) {
            taskQueueFactory = (EventLoopTaskQueueFactory) args[3];
        }
        if (argsLength > 4) {
            tailTaskQueueFactory = (EventLoopTaskQueueFactory) args[4];
        }
        return new EpollEventLoop(this, executor, maxEvents,
                selectStrategyFactory.newSelectStrategy(),
                rejectedExecutionHandler, taskQueueFactory, tailTaskQueueFactory);
    }

As can be seen from the method, newChild accepts an executor and multiple additional parameters, which are SelectStrategyFactory, RejectedExecutionHandler, taskQueueFactory and tailTaskQueueFactory, and finally pass these parameters into EpollEventLoop and return a new EpollEventLoop object.

EpollEventLoop

EpollEventLoop is created by EpollEventLoopGroup using the new child method.

For EpollEventLoop itself, it is a SingleThreadEventLoop:

 class EpollEventLoop extends SingleThreadEventLoop

With the powerful functions of native epoll IO, EpollEventLoop can quickly perform business processing in a single-threaded situation, which is very good.

Like EpollEventLoopGroup, EpollEventLoop needs to check whether the system supports epoll during initialization:

 static {
        Epoll.ensureAvailability();
    }

In the constructor of EpollEventLoop called by EpollEventLoopGroup, three FileDescriptors are initialized, namely epollFd, eventFd and timerFd. These three FileDescriptors are created by calling the Native method:

 this.epollFd = epollFd = Native.newEpollCreate();
this.eventFd = eventFd = Native.newEventFd();
this.timerFd = timerFd = Native.newTimerFd();

Then call Native.epollCtlAdd to establish the association between FileDescriptors:

 Native.epollCtlAdd(epollFd.intValue(), eventFd.intValue(), Native.EPOLLIN | Native.EPOLLET);
Native.epollCtlAdd(epollFd.intValue(), timerFd.intValue(), Native.EPOLLIN | Native.EPOLLET);

In the run method of EpollEventLoop, the selectStrategy.calculateStrategy method will be called first to get the current select state. By default, there are three states, namely:

 int SELECT = -1;

    int CONTINUE = -2;

    int BUSY_WAIT = -3;

We have introduced these three states in kqueue. The difference is that epoll supports the BUSY_WAIT state. In the BUSY_WAIT state, it will call the Native.epollBusyWait(epollFd, events) method to return the number of events of busy wait.

If it is in the select state, it will call the Native.epollWait(epollFd, events, 1000) method to return the number of events in the wait state.

Next, the processReady(events, strategy) and runAllTasks methods will be called respectively to perform the event's ready state callback processing and final task execution.

EpollServerSocketChannel

First look at the definition of EpollServerSocketChannel:

 public final class EpollServerSocketChannel extends AbstractEpollServerChannel implements ServerSocketChannel

EpollServerSocketChannel inherits from AbstractEpollServerChannel and implements the ServerSocketChannel interface.

The constructor of EpollServerSocketChannel needs to pass in a LinuxSocket:

 EpollServerSocketChannel(LinuxSocket fd) {
        super(fd);
        config = new EpollServerSocketChannelConfig(this);
    }

LinuxSocket is a special socket used to handle native socket connections with linux.

EpollServerSocketChannelConfig is the configuration for building EpollServerSocketChannel. Four configuration options are used here, namely SO_REUSEPORT, IP_FREEBIND, IP_TRANSPARENT, TCP_DEFER_ACCEPT and TCP_MD5SIG. Each configuration item corresponds to the specific meaning of the network protocol.

Let's take another look at the newChildChannel method of EpollServerSocketChannel:

 protected Channel newChildChannel(int fd, byte[] address, int offset, int len) throws Exception {
        return new EpollSocketChannel(this, new LinuxSocket(fd), address(address, offset, len));
    }

Like the KqueueServerSocketChannel method, newChildChannel also returns an EpollSocketChannel and constructs the incoming fd into a LinuxSocket.

EpollSocketChannel

EpollSocketChannel is created and returned by EpollServerSocketChannel, first look at the definition of EpollSocketChannel:

 public final class EpollSocketChannel extends AbstractEpollStreamChannel implements SocketChannel {

You can see that EpollSocketChannel inherits from AbstractEpollStreamChannel and implements the SocketChannel interface.

Going back to the newChildChannel method called when EpollServerSocketChannel created EpollSocketChannel before, this method will call the constructor of EpollSocketChannel as follows:

 EpollSocketChannel(Channel parent, LinuxSocket fd, InetSocketAddress remoteAddress) {
        super(parent, fd, remoteAddress);
        config = new EpollSocketChannelConfig(this);

        if (parent instanceof EpollServerSocketChannel) {
            tcpMd5SigAddresses = ((EpollServerSocketChannel) parent).tcpMd5SigAddresses();
        }
    }

As can be seen from the logic of the code, if the EpollSocketChannel is created from the EpollServerSocketChannel, the tcpMd5Sig feature will be enabled by default.

What is tcpMd5Sig?

To put it simply, tcpMd5Sig is to add MD5 sig to the TCP data message, which is used to verify the data, thus indicating the security of data transmission.

TCP MD5 is proposed in RFC 2385 and can only be turned on in the linux kernel, which means that if you want to use tcpMd5Sig, you must use EpollServerSocketChannel and EpollSocketChannel.

Therefore, if you are a friend who pursues performance or special usage scenarios, there are still many times when you need to contact this kind of native transport, and you can study the configuration options carefully.

Take another look at the very important doConnect0 method in EpollSocketChannel:

 boolean doConnect0(SocketAddress remote) throws Exception {
        if (IS_SUPPORTING_TCP_FASTOPEN_CLIENT && config.isTcpFastOpenConnect()) {
            ChannelOutboundBuffer outbound = unsafe().outboundBuffer();
            outbound.addFlush();
            Object curr;
            if ((curr = outbound.current()) instanceof ByteBuf) {
                ByteBuf initialData = (ByteBuf) curr;
                long localFlushedAmount = doWriteOrSendBytes(
                        initialData, (InetSocketAddress) remote, true);
                if (localFlushedAmount > 0) {
                    outbound.removeBytes(localFlushedAmount);
                    return true;
                }
            }
        }
        return super.doConnect0(remote);
    }

In this method, it will first determine whether the TcpFastOpen option is enabled. If this option is enabled, the write or sendTo method of LinuxSocket will eventually be called. These methods can add initial data and transmit data while establishing a connection, so as to achieve Tcp fast The effect of open.

If it is not tcp fast open, then you need to call the connect method of Socket to establish a traditional connection.

Summarize

The implementation of epoll in netty is very similar to kqueue. Their difference lies in the running platform and specific functional parameters. If you are pursuing high performance, you can study it in depth.

The code of this article, you can refer to:

learn-netty4

For more information, please refer to http://www.flydean.com/53-2-netty-epoll-transport/

The most popular interpretation, the most profound dry goods, the most concise tutorials, and many tricks you don't know are waiting for you to discover!

Welcome to pay attention to my official account: "Program those things", understand technology, understand you better!