Disruptor notes five: event consumption actual combat

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

"Disruptor Notes" series link

1. Quick Start
2. Disruptor class analysis
3. Basic operation of circular queue (without Disruptor class)
4. Summary of event consumption knowledge points
5. event consumption actual combat
6. Common scenarios
7. Waiting strategy
8. Knowledge Points Supplement (Final)

Overview of this article

This is the fifth chapter of "Disruptor Notes". The previous article "Disruptor Notes IV: Summary of Knowledge Points on Event Consumption" theoretically combs and analyzes independent consumption and joint consumption, leaving three tasks, which will be completed today. These tasks, namely coding, realize the following three scenarios:

1. 100 orders, independent consumption by SMS and mail system
2. 100 orders, the two mail servers of the mail system consume together;
3. 100 orders, the SMS system consumes independently, at the same time, the two mail servers consume together;

Source download

• The complete source code of this actual combat can be downloaded from GitHub. The address and link information are shown in the following table ( https://github.com/zq2599/blog_demos):

name	Link	Remark
Project homepage	https://github.com/zq2599/blog_demos	The project's homepage on GitHub
git warehouse address (https)	https://github.com/zq2599/blog_demos.git	The warehouse address of the source code of the project, https protocol
git warehouse address (ssh)	[e-mail]:zq2599/blog_demos.git	The warehouse address of the source code of the project, ssh protocol

• There are multiple folders in this git project. The source code of this actual combat is under the <font color="blue">disruptor-tutorials</font> folder, as shown in the red box in the following figure:

• <font color="blue">disruptor-tutorials</font> is a parent project with multiple modules. The actual module in this article is <font color="red">consume-mode</font>, as shown in the red below The box shows:

Write common code

• In order to complete the task and code to realize the above three scenarios, we need to write the public code first;
• First, create a new module named <font color="red">consume-mode</font> under the parent project <font color="blue">disruptor-tutorials</font>, and its build.gradle content is as follows:

plugins {
    id 'org.springframework.boot'
}

dependencies {
    implementation 'org.projectlombok:lombok'
    implementation 'org.springframework.boot:spring-boot-starter'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'com.lmax:disruptor'

    testImplementation('org.springframework.boot:spring-boot-starter-test')
}

• Springboot startup class:

package com.bolingcavalry;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ConsumeModeApplication {
    public static void main(String[] args) {
        SpringApplication.run(ConsumeModeApplication.class, args);
    }
}

• Order event definition:

package com.bolingcavalry.service;

import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.ToString;

@Data
@ToString
@NoArgsConstructor
public class OrderEvent {

    private String value;
}

• • The engineering class of the order event, which defines how to create the event instance:

package com.bolingcavalry.service;

import com.lmax.disruptor.EventFactory;

public class OrderEventFactory implements EventFactory<OrderEvent> {

    @Override
    public OrderEvent newInstance() {
        return new OrderEvent();
    }
}

• The order event producer class defines how to publish business information to the circular queue through events:

package com.bolingcavalry.service;

import com.lmax.disruptor.RingBuffer;

public class OrderEventProducer {
    // 存储数据的环形队列
    private final RingBuffer<OrderEvent> ringBuffer;

    public OrderEventProducer(RingBuffer<OrderEvent> ringBuffer) {
        this.ringBuffer = ringBuffer;
    }

    public void onData(String content) {
        // ringBuffer是个队列，其next方法返回的是下最后一条记录之后的位置，这是个可用位置
        long sequence = ringBuffer.next();

        try {
            // sequence位置取出的事件是空事件
            OrderEvent orderEvent = ringBuffer.get(sequence);
            // 空事件添加业务信息
            orderEvent.setValue(content);
        } finally {
            // 发布
            ringBuffer.publish(sequence);
        }
    }
}

• The SMS service for consuming order events implements the EventHandler interface, so it is used in independently consuming </font>:

package com.bolingcavalry.service;

import com.lmax.disruptor.EventHandler;
import lombok.extern.slf4j.Slf4j;
import java.util.function.Consumer;

@Slf4j
public class SmsEventHandler implements EventHandler<OrderEvent> {

    public SmsEventHandler(Consumer<?> consumer) {
        this.consumer = consumer;
    }

    // 外部可以传入Consumer实现类，每处理一条消息的时候，consumer的accept方法就会被执行一次
    private Consumer<?> consumer;

    @Override
    public void onEvent(OrderEvent event, long sequence, boolean endOfBatch) throws Exception {
        log.info("短信服务 sequence [{}], endOfBatch [{}], event : {}", sequence, endOfBatch, event);

        // 这里延时100ms，模拟消费事件的逻辑的耗时
        Thread.sleep(100);

        // 如果外部传入了consumer，就要执行一次accept方法
        if (null!=consumer) {
            consumer.accept(null);
        }
    }
}

• The mail service for consuming order events implements the EventHandler interface, so it is used in independently consuming </font>:

package com.bolingcavalry.service;

import com.lmax.disruptor.EventHandler;
import lombok.extern.slf4j.Slf4j;
import java.util.function.Consumer;

@Slf4j
public class MailEventHandler implements EventHandler<OrderEvent> {

    public MailEventHandler(Consumer<?> consumer) {
        this.consumer = consumer;
    }

    // 外部可以传入Consumer实现类，每处理一条消息的时候，consumer的accept方法就会被执行一次
    private Consumer<?> consumer;

    @Override
    public void onEvent(OrderEvent event, long sequence, boolean endOfBatch) throws Exception {
        log.info("邮件服务 sequence [{}], endOfBatch [{}], event : {}", sequence, endOfBatch, event);

        // 这里延时100ms，模拟消费事件的逻辑的耗时
        Thread.sleep(100);

        // 如果外部传入了consumer，就要执行一次accept方法
        if (null!=consumer) {
            consumer.accept(null);
        }
    }
}

• The mail service for consuming order events implements the WorkHandler interface, so it is used in co-consuming </font>:

package com.bolingcavalry.service;

import com.lmax.disruptor.WorkHandler;
import lombok.extern.slf4j.Slf4j;
import java.util.function.Consumer;

@Slf4j
public class MailWorkHandler implements WorkHandler<OrderEvent> {

    public MailWorkHandler(Consumer<?> consumer) {
        this.consumer = consumer;
    }

    // 外部可以传入Consumer实现类，每处理一条消息的时候，consumer的accept方法就会被执行一次
    private Consumer<?> consumer;

    @Override
    public void onEvent(OrderEvent event) throws Exception {
        log.info("共同消费模式的邮件服务 : {}", event);

        // 这里延时100ms，模拟消费事件的逻辑的耗时
        Thread.sleep(100);

        // 如果外部传入了consumer，就要执行一次accept方法
        if (null!=consumer) {
            consumer.accept(null);
        }
    }
}

• Finally, the logic of publishing and consuming events is written in an abstract class, but how to consume events is not implemented in this class, but is reserved for subclasses. In this abstract class, there are several points to pay attention to later. arrive:

package com.bolingcavalry.service;

import com.lmax.disruptor.dsl.Disruptor;
import lombok.Setter;
import org.springframework.scheduling.concurrent.CustomizableThreadFactory;
import javax.annotation.PostConstruct;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;

public abstract class ConsumeModeService {
    /**
     * 独立消费者数量
     */
    public static final int INDEPENDENT_CONSUMER_NUM = 2;

    /**
     * 环形缓冲区大小
     */
    protected int BUFFER_SIZE = 16;

    protected Disruptor<OrderEvent> disruptor;

    @Setter
    private OrderEventProducer producer;

    /**
     * 统计消息总数
     */
    protected final AtomicLong eventCount = new AtomicLong();

    /**
     * 这是辅助测试用的，
     * 测试的时候，完成事件发布后，测试主线程就用这个countDownLatch开始等待，
     * 在消费到指定的数量(countDownLatchGate)后，消费线程执行countDownLatch的countDown方法，
     * 这样测试主线程就可以结束等待了
     */
    private CountDownLatch countDownLatch;

    /**
     * 这是辅助测试用的，
     * 测试的时候，完成事件发布后，测试主线程就用这个countDownLatch开始等待，
     * 在消费到指定的数量(countDownLatchGate)后，消费线程执行countDownLatch的countDown方法，
     * 这样测试主线程就可以结束等待了
     */
    private int countDownLatchGate;

    /**
     * 准备一个匿名类，传给disruptor的事件处理类，
     * 这样每次处理事件时，都会将已经处理事件的总数打印出来
     */
    protected Consumer<?> eventCountPrinter = new Consumer<Object>() {
        @Override
        public void accept(Object o) {
            long count = eventCount.incrementAndGet();

            /**
             * 这是辅助测试用的，
             * 测试的时候，完成事件发布后，测试主线程就用这个countDownLatch开始等待，
             * 在消费到指定的数量(countDownLatchGate)后，消费线程执行countDownLatch的countDown方法，
             * 这样测试主线程就可以结束等待了
             */
            if (null!=countDownLatch && count>=countDownLatchGate) {
                countDownLatch.countDown();
            }
        }
    };

    /**
     * 发布一个事件
     * @param value
     * @return
     */
    public void publish(String value) {
        producer.onData(value);
    }

    /**
     * 返回已经处理的任务总数
     * @return
     */
    public long eventCount() {
        return eventCount.get();
    }

    /**
     * 这是辅助测试用的，
     * 测试的时候，完成事件发布后，测试主线程就用这个countDownLatch开始等待，
     * 在消费到指定的数量(countDownLatchGate)后，消费线程执行countDownLatch的countDown方法，
     * 这样测试主线程就可以结束等待了
     * @param countDownLatch
     * @param countDownLatchGate
     */
    public void setCountDown(CountDownLatch countDownLatch, int countDownLatchGate) {
        this.countDownLatch = countDownLatch;
        this.countDownLatchGate = countDownLatchGate;
    }

    /**
     * 留给子类实现具体的事件消费逻辑
     */
    protected abstract void disruptorOperate();

    @PostConstruct
    private void init() {
        // 实例化
        disruptor = new Disruptor<>(new OrderEventFactory(),
                BUFFER_SIZE,
                new CustomizableThreadFactory("event-handler-"));

        // 留给子类实现具体的事件消费逻辑
        disruptorOperate();

        // 启动
        disruptor.start();

        // 生产者
        setProducer(new OrderEventProducer(disruptor.getRingBuffer()));
    }
}

• The above code, there are the following points to note:
• The init method is the method to be executed after the spring bean is instantiated. The Disruptor is instantiated, the consumer thread is started, and the event producer is instantiated. The specific event consumption logic is implemented by the subclass in the disruptorOperate method;
• eventCountPrinter is an anonymous class instance. After it is passed to the event consumption handler, the eventCountPrinter.accept method will be executed every time an event is consumed, so that the total number of consumption events is accurately stored in the eventCount variable;
• CountDownLatch and countDownLatchGate are prepared to assist unit testing. During testing, after the event is released, the main test thread uses this countDownLatch to start waiting. After the specified amount (countDownLatchGate) is consumed, the consumer thread executes the countDown method of countDownLatch. In this way, the main thread of the test can end the waiting
• At this point, the common code is finished. It can be seen that the abstract parent class has done most of the things. Our subclasses can focus on the logical arrangement of event consumption, and start to implement the three scenarios one by one;

100 orders, independent consumption by SMS and mail system

• The logic of independent consumption by two consumers is very simple, just one line of code, call the <font color="blue">handleEventsWith</font> method to pass in all consumer instances, and it's done:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.ConsumeModeService;
import com.bolingcavalry.service.MailEventHandler;
import com.bolingcavalry.service.SmsEventHandler;
import org.springframework.stereotype.Service;

@Service("independentModeService")
public class IndependentModeServiceImpl extends ConsumeModeService {

    @Override
    protected void disruptorOperate() {
        // 调用handleEventsWith，表示创建的多个消费者，每个都是独立消费的
        // 这里创建两个消费者，一个是短信的，一个是邮件的
        disruptor.handleEventsWith(new SmsEventHandler(eventCountPrinter), new MailEventHandler(eventCountPrinter));
    }
}

• The unit test code is as follows, the place to be noted is that after publishing the <font color="red"> 100 </font> event, call the <font color="blue">countDownLatch.await()</font> method to start Wait until the consumer thread calls the <font color="blue">countDownLatch.countDown()</font> method to release the wait, and the total number of expected consumption messages is equal to <font color="red"> 200 </font> font>:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.ConsumeModeService;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import java.util.concurrent.CountDownLatch;
import static org.junit.Assert.assertEquals;

@RunWith(SpringRunner.class)
@SpringBootTest
@Slf4j
public class ConsumeModeServiceTest {

    @Autowired
    @Qualifier("independentModeService")
    ConsumeModeService independentModeService;

    /**
     * 测试时生产的消息数量
     */
    private static final int EVENT_COUNT = 100;

    private void testConsumeModeService(ConsumeModeService service, int eventCount, int expectEventCount) throws InterruptedException {
        CountDownLatch countDownLatch = new CountDownLatch(1);

        // 告诉service，等消费到expectEventCount个消息时，就执行countDownLatch.countDown方法
        service.setCountDown(countDownLatch, expectEventCount);

        for(int i=0;i<eventCount;i++) {
            log.info("publich {}", i);
            service.publish(String.valueOf(i));
        }

        // 当前线程开始等待，前面的service.setCountDown方法已经告诉过service，
        // 等消费到expectEventCount个消息时，就执行countDownLatch.countDown方法
        // 千万注意，要调用await方法，而不是wait方法！
        countDownLatch.await();

        // 消费的事件总数应该等于发布的事件数
        assertEquals(expectEventCount, service.eventCount());
    }

    @Test
    public void testIndependentModeService() throws InterruptedException {
        log.info("start testIndependentModeService");
        testConsumeModeService(independentModeService,
                EVENT_COUNT,
                EVENT_COUNT * ConsumeModeService.INDEPENDENT_CONSUMER_NUM);
    }
}

• The unit test execution results are as follows, in line with expectations:

100 orders, the two mail servers of the mail system consume together

• The code for two consumers to consume together is also very simple, just call the handleEventsWithWorkerPool method, and pass in the MailWorkHandler instance that is consumed together as a parameter:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.ConsumeModeService;
import com.bolingcavalry.service.MailWorkHandler;
import org.springframework.stereotype.Service;

@Service("shareModeService")
public class ShareModeServiceImpl extends ConsumeModeService {
    @Override
    protected void disruptorOperate() {
        // mailWorkHandler1模拟一号邮件服务器
        MailWorkHandler mailWorkHandler1 = new MailWorkHandler(eventCountPrinter);

        // mailWorkHandler2模拟一号邮件服务器
        MailWorkHandler mailWorkHandler2 = new MailWorkHandler(eventCountPrinter);

        // 调用handleEventsWithWorkerPool，表示创建的多个消费者以共同消费的模式消费
        disruptor.handleEventsWithWorkerPool(mailWorkHandler1, mailWorkHandler2);
    }
}

• The unit test is to add the following code in ConsumeModeServiceTest.java. Note that since it is a common consumption, the expected number of consumption events is equal to the number of messages, which is 100:

    @Autowired
    @Qualifier("shareModeService")
    ConsumeModeService shareModeService;

    @Test
    public void testShareModeService() throws InterruptedException {
        log.info("start testShareModeService");
        testConsumeModeService(shareModeService, EVENT_COUNT, EVENT_COUNT);
    }

• Perform unit test, the result is as follows:

100 orders, the SMS system consumes independently, at the same time, the two mail servers consume together

• The last scene is still very simple, call handleEventsWith once, then call handleEventsWithWorkerPool again:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.ConsumeModeService;
import com.bolingcavalry.service.MailWorkHandler;
import com.bolingcavalry.service.SmsEventHandler;
import org.springframework.stereotype.Service;

@Service("independentAndShareModeService")
public class IndependentAndShareModeServiceImpl extends ConsumeModeService {
    @Override
    protected void disruptorOperate() {
        // 调用handleEventsWith，表示创建的多个消费者，每个都是独立消费的
        // 这里创建一个消费者，短信服务
        disruptor.handleEventsWith(new SmsEventHandler(eventCountPrinter));

        // mailWorkHandler1模拟一号邮件服务器
        MailWorkHandler mailWorkHandler1 = new MailWorkHandler(eventCountPrinter);

        // mailWorkHandler2模拟一号邮件服务器
        MailWorkHandler mailWorkHandler2 = new MailWorkHandler(eventCountPrinter);

        // 调用handleEventsWithWorkerPool，表示创建的多个消费者以共同消费的模式消费
        disruptor.handleEventsWithWorkerPool(mailWorkHandler1, mailWorkHandler2);
    }
}

• The unit test is to add the following code to ConsumeModeServiceTest.java, the expected number of consumption events should be 200, because there are two independent consumptions as a whole, but one of them has two consumers inside:

    @Autowired
    @Qualifier("independentAndShareModeService")
    ConsumeModeService independentAndShareModeService;

    @Test
    public void independentAndShareModeService() throws InterruptedException {
        log.info("start independentAndShareModeService");
        testConsumeModeService(independentAndShareModeService,
                EVENT_COUNT,
                EVENT_COUNT * ConsumeModeService.INDEPENDENT_CONSUMER_NUM);
    }

• The unit test results are as follows, in line with expectations:

• At this point, the actual combat of independent consumption and joint consumption is completed. With the help of disruptor, the three common scenarios can be easily completed. If you are doing the development of these scenarios, I hope this article can give you some reference;

You are not alone, Xinchen and original are with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. Database + Middleware Series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos