Kafka source code analysis: the running process of the server

Abstract: The server side of Kafka network module, introduces the server side to start, receive requests and process requests.

This article is shared from Huawei Cloud Community " Kafka Network Module-Server Side ", the original author: middleware brother.

SocketServer is a module used to process requests on the Kafka server. It is created, initialized, and started during the Kafka startup process.

SocketServer startup process:

• Initialize the Acceptor in the order of endpoints, each endpoint corresponds to an Acceptor, create a Processor for each Acceptor (the number is determined by the num.network.threads configuration item), and start the Acceptor. After the Acceptor is started, the connection will be monitored through the selector and the newly established The connection is handed over to the Processor for processing (polling to select Processor)
• Start all processors
  Acceptor starts and monitors the connection process:
  
• After Acceptor starts, it will create a serverSocketChannel, monitor on the endpoint corresponding to the acceptor, and register OP_ACCEPT on the selector, and then enter an endless loop. In each cycle, the ready key (that is, the previously registered serverSocketChannel) is obtained through the selector, indicating that there is The connection comes, and then create a socketChannel corresponding to the connection through accept(), and then poll to select one from the processors in charge of the acceptor, and hand the socketChannel to the selected processor for processing, that is, the connection is handed over to the processor.
• Acceptor hands the connection to the processor for processing. It adds the socketChannel to the processor's connection queue newConnection, and the processor will continuously obtain and process it in the run method.
• After the processor obtains the socketChannel from newConnection, it registers OP_READ on the selector and creates the corresponding KafkaChannel.

Server receiving request and processing process:

• After the processor receives the OP_READ event and is ready, it checks and tries to complete the SSL handshake and SASL verification (the handshake may not be completed at this time, so after the processor receives the OP_READ event, it must first check and ensure that the handshake has been completed, SSL/SASL (Related reference section 9.4)
• After the SSL handshake and SASL verification are completed, read the data from the channel, construct the NetworkReceive object, and merge it into the stagedReceives team
• Take out the first element of stagedReceives (remove) and add completedReceives
• Take out the elements in completedReceives (not remove), construct a Request object, add it to requestQueue, remove the event registration for OP_READ, and set the corresponding KafkaChannel to MUTED, and then to MUTED_AND_RESPONSE_PENDING
• KafkaRequestHandler fetches elements from requestQueue (removed) and hands them to KafkaApi module to process the request
• After KafkaApi processes the request, it puts the response into the responseQueue and inflightResponses of the corresponding processor, and wakes up its selector
• The Processor takes the response (removed) from the responseQueue. If the response needs to be sent back to the client, it assigns the response send to KafkaChannel and registers the OP_WRITE event
• When the channel is ready, write send to the write buffer of the channel. When send is finished, remove the registration of OP_WRITE event and add send to completedSends
• Remove the corresponding response from inflightResponses, execute the response callback, set KafkaChannel to MUTED, then set MUTED to NOT_MUTED, and re-add the OP_READ event registration

Click to follow, and get to know the fresh technology of Huawei Cloud for the first time~