[mq] Implement mq-01-producer and consumer startup from scratch

What is MQ?

MQ (Message Queue) message queue is a data structure of "first in, first out" in the basic data structure.

Refers to putting the data (message) to be transmitted in the queue, and using the queue mechanism to achieve message delivery - the producer generates the message and puts the message into the queue, and then the consumer handles it.

Consumers can pull messages from the specified queue, or subscribe to the corresponding queue, and the MQ server can push messages to it.

What is the role of MQ?

The message queue middleware is an important component in the distributed system. It mainly solves problems such as application decoupling, asynchronous messages, and traffic cutting, and realizes a high-performance, high-availability, scalable and eventually consistent architecture.

Decoupling: A business needs to be implemented by multiple modules, or a message needs to be processed by multiple systems. It only needs to send an MQ after the main business is completed, and the other modules consume MQ messages to realize the business and reduce the communication between modules. coupling.

Asynchronous: After the main business is executed, the subordinate business is executed asynchronously through MQ, which reduces the response time of the business and improves the user experience.

Peak shaving: In the case of high concurrency, business is processed asynchronously, providing business processing capabilities during peak periods and avoiding system paralysis.

ps: The above content is excerpted from Wikipedia.

Preparations for implementing mq

maven import

 <dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-all</artifactId>
    <version>4.1.42.Final</version>
</dependency>

<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.76</version>
</dependency>

Module division

The message queue in java. As a zero-based learning project of mq, it is currently open source.

The modules of the project are as follows:

module	illustrate
mq-common	public code
mq-broker	Registration Center
mq-producer	message producer
mq-consumer	message consumer

message consumer

Interface definition

 package com.github.houbb.mq.consumer.api;

/**
 * @author binbin.hou
 * @since 1.0.0
 */
public interface IMqConsumer {

    /**
     * 订阅
     * @param topicName topic 名称
     * @param tagRegex 标签正则
     */
    void subscribe(String topicName, String tagRegex);

    /**
     * 注册监听器
     * @param listener 监听器
     */
    void registerListener(final IMqConsumerListener listener);

}

IMqConsumerListener As the interface of the message monitoring class, it is defined as follows:

 public interface IMqConsumerListener {


    /**
     * 消费
     * @param mqMessage 消息体
     * @param context 上下文
     * @return 结果
     */
    ConsumerStatus consumer(final MqMessage mqMessage,
                            final IMqConsumerListenerContext context);

}

ConsumerStatus represents several states of message consumption.

message body

The startup message body MqMessage is defined as follows:

 package com.github.houbb.mq.common.dto;

import java.util.Arrays;
import java.util.List;

/**
 * @author binbin.hou
 * @since 1.0.0
 */
public class MqMessage {

    /**
     * 标题名称
     */
    private String topic;

    /**
     * 标签
     */
    private List<String> tags;

    /**
     * 内容
     */
    private byte[] payload;

    /**
     * 业务标识
     */
    private String bizKey;

    /**
     * 负载分片标识
     */
    private String shardingKey;

    // getter&setter&toString

}

push consumer strategy implementation

The implementation of consumer initiation is as follows:

 /**
 * 推送消费策略
 *
 * @author binbin.hou
 * @since 1.0.0
 */
public class MqConsumerPush extends Thread implements IMqConsumer  {

    // 省略...

    @Override
    public void run() {
        // 启动服务端
        log.info("MQ 消费者开始启动服务端 groupName: {}, port: {}, brokerAddress: {}",
                groupName, port, brokerAddress);

        EventLoopGroup bossGroup = new NioEventLoopGroup();
        EventLoopGroup workerGroup = new NioEventLoopGroup();

        try {
            ServerBootstrap serverBootstrap = new ServerBootstrap();
            serverBootstrap.group(workerGroup, bossGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<Channel>() {
                        @Override
                        protected void initChannel(Channel ch) throws Exception {
                            ch.pipeline().addLast(new MqConsumerHandler());
                        }
                    })
                    // 这个参数影响的是还没有被accept 取出的连接
                    .option(ChannelOption.SO_BACKLOG, 128)
                    // 这个参数只是过一段时间内客户端没有响应，服务端会发送一个 ack 包，以判断客户端是否还活着。
                    .childOption(ChannelOption.SO_KEEPALIVE, true);

            // 绑定端口，开始接收进来的链接
            ChannelFuture channelFuture = serverBootstrap.bind(port).syncUninterruptibly();
            log.info("MQ 消费者启动完成，监听【" + port + "】端口");

            channelFuture.channel().closeFuture().syncUninterruptibly();
            log.info("MQ 消费者关闭完成");
        } catch (Exception e) {
            log.error("MQ 消费者启动异常", e);
            throw new MqException(ConsumerRespCode.RPC_INIT_FAILED);
        } finally {
            workerGroup.shutdownGracefully();
            bossGroup.shutdownGracefully();
        }


    }


    // 省略...

}

ps: In the early days, we used the consumer as the server, and when the broker was introduced later, only the broker was the server.

MqConsumerHandler processing class

This class is an empty implementation.

 public class MqConsumerHandler extends SimpleChannelInboundHandler {

    @Override
    protected void channelRead0(ChannelHandlerContext channelHandlerContext, Object object) throws Exception {
        //nothing
    }

}

test code

 MqConsumerPush mqConsumerPush = new MqConsumerPush();
mqConsumerPush.start();

Startup log:

 [DEBUG] [2022-04-21 19:16:41.343] [main] [c.g.h.l.i.c.LogFactory.setImplementation] - Logging initialized using 'class com.github.houbb.log.integration.adaptors.stdout.StdOutExImpl' adapter.
[INFO] [2022-04-21 19:16:41.356] [Thread-0] [c.g.h.m.c.c.MqConsumerPush.run] - MQ 消费者开始启动服务端 groupName: C_DEFAULT_GROUP_NAME, port: 9527, brokerAddress: 
[INFO] [2022-04-21 19:16:43.196] [Thread-0] [c.g.h.m.c.c.MqConsumerPush.run] - MQ 消费者启动完成，监听【9527】端口

message producer

Interface definition

The most basic message sending interface.

 package com.github.houbb.mq.producer.api;

import com.github.houbb.mq.common.dto.MqMessage;
import com.github.houbb.mq.producer.dto.SendResult;

/**
 * @author binbin.hou
 * @since 1.0.0
 */
public interface IMqProducer {

    /**
     * 同步发送消息
     * @param mqMessage 消息类型
     * @return 结果
     */
    SendResult send(final MqMessage mqMessage);

    /**
     * 单向发送消息
     * @param mqMessage 消息类型
     * @return 结果
     */
    SendResult sendOneWay(final MqMessage mqMessage);

}

Producer implementation

The implementation of MqProducer startup is as follows, based on netty.

 package com.github.houbb.mq.producer.core;

/**
 * 默认 mq 生产者
 * @author binbin.hou
 * @since 1.0.0
 */
public class MqProducer extends Thread implements IMqProducer {

    //省略...

    @Override
    public void run() {
        // 启动服务端
        log.info("MQ 生产者开始启动客户端 GROUP: {}, PORT: {}, brokerAddress: {}",
                groupName, port, brokerAddress);

        EventLoopGroup workerGroup = new NioEventLoopGroup();

        try {
            Bootstrap bootstrap = new Bootstrap();
            ChannelFuture channelFuture = bootstrap.group(workerGroup)
                    .channel(NioSocketChannel.class)
                    .option(ChannelOption.SO_KEEPALIVE, true)
                    .handler(new ChannelInitializer<Channel>(){
                        @Override
                        protected void initChannel(Channel ch) throws Exception {
                            ch.pipeline()
                                    .addLast(new LoggingHandler(LogLevel.INFO))
                                    .addLast(new MqProducerHandler());
                        }
                    })
                    .connect("localhost", port)
                    .syncUninterruptibly();

            log.info("MQ 生产者启动客户端完成，监听端口：" + port);
            channelFuture.channel().closeFuture().syncUninterruptibly();
            log.info("MQ 生产者开始客户端已关闭");
        } catch (Exception e) {
            log.error("MQ 生产者启动遇到异常", e);
            throw new MqException(ProducerRespCode.RPC_INIT_FAILED);
        } finally {
            workerGroup.shutdownGracefully();
        }
    }

    //省略...
}

MqProducerHandler processing class

The default empty implementation does nothing.

 package com.github.houbb.mq.producer.handler;

import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;

/**
 * @author binbin.hou
 * @since 1.0.0
 */
public class MqProducerHandler extends SimpleChannelInboundHandler {

    @Override
    protected void channelRead0(ChannelHandlerContext channelHandlerContext, Object object) throws Exception {
        //do nothing now
    }

}

startup code

 MqProducer mqProducer = new MqProducer();
mqProducer.start();

Startup log:

 [DEBUG] [2022-04-21 19:17:11.960] [main] [c.g.h.l.i.c.LogFactory.setImplementation] - Logging initialized using 'class com.github.houbb.log.integration.adaptors.stdout.StdOutExImpl' adapter.
[INFO] [2022-04-21 19:17:11.974] [Thread-0] [c.g.h.m.p.c.MqProducer.run] - MQ 生产者开始启动客户端 GROUP: P_DEFAULT_GROUP_NAME, PORT: 9527, brokerAddress: 
四月 21, 2022 7:17:13 下午 io.netty.handler.logging.LoggingHandler channelRegistered
信息: [id: 0x5cb48145] REGISTERED
四月 21, 2022 7:17:13 下午 io.netty.handler.logging.LoggingHandler connect
信息: [id: 0x5cb48145] CONNECT: localhost/127.0.0.1:9527
四月 21, 2022 7:17:13 下午 io.netty.handler.logging.LoggingHandler channelActive
信息: [id: 0x5cb48145, L:/127.0.0.1:57740 - R:localhost/127.0.0.1:9527] ACTIVE
[INFO] [2022-04-21 19:17:13.833] [Thread-0] [c.g.h.m.p.c.MqProducer.run] - MQ 生产者启动客户端完成，监听端口：9527

summary

Based on netty, the most basic server startup and client startup are over here.

A thousand miles begins with a single step.

We will learn with you in the next section how to realize the interaction between the client and the server.

I hope this article is helpful to you. If you like it, please like, collect and forward it.

I'm an old horse, and I look forward to seeing you again next time.

open source address

The message queue in java.(java simple version mq implementation) : https://github.com/houbb/mq

Extended reading

rpc - Implementing rpc from scratch : https://github.com/houbb/rpc

[mq] Implement mq-01-producer and consumer startup from scratch