Disruptor note three: basic operation of circular queue (without Disruptor class)

Welcome to my GitHub

https://github.com/zq2599/blog_demos

Content: Classification and summary of all original articles and supporting source code, involving Java, Docker, Kubernetes, DevOPS, etc.;

"Disruptor Notes" series link

1. Quick Start
2. Disruptor class analysis
3. circular queue (without Disruptor class)
4. Summary of event consumption knowledge points
5. event consumption actual combat
6. Common scenarios
7. Waiting strategy
8. Knowledge Points (Final)

Overview of this article

• This article is the third in the series of "Disruptor Notes". The main task is to code to achieve message production and consumption. The difference from "One of the disruptor notes: Quick Start" is that this development does not use the Disruptor class, and the Ring Buffer( (Circular queue) related operations are implemented by writing code by yourself;
• This method of operating Ring Buffer out of the Disruptor class is not suitable for use in a production environment, but in the process of learning Disruptor, this is an efficient learning method. After the actual combat in this article, when using Disruptor in the future, you are developing, It can be more handy in various scenarios such as debugging and optimization;
• Simple news production and consumption can no longer satisfy our enthusiasm for learning. Today's actual combat has to challenge the following three scenarios:
• 100 events, single consumer consumption;
• 100 events, three consumers, each consumes the 100 events alone;
• For 100 events, three consumers consume the 100 events together;

Previous review

In order to complete the actual combat of this article, the previous article "Disruptor Notes II: Disruptor Class Analysis" has been fully researched and analyzed, and it is recommended to watch, here is a brief review of the following core functions of the Disruptor class, which is also what we need to achieve when coding of:

1. Create a ring queue (RingBuffer object)
2. Create SequenceBarrier object to receive consumable events in ringBuffer
3. Create BatchEventProcessor, responsible for consuming events
4. Binding the exception handling class of the BatchEventProcessor object
5. Call ringBuffer.addGatingSequences to pass the Consumer's Sequence to ringBuffer
6. Start an independent thread to execute the business logic of the consumption event
7. The theoretical analysis has been completed, and then start coding;

Source download

• The complete source code in this actual combat can be downloaded on GitHub, the address and link information are shown in the following table ( https://github.com/zq2599/blog_demos):

name	Link	Remark
Project homepage	https://github.com/zq2599/blog_demos	The project's homepage on GitHub
git warehouse address (https)	https://github.com/zq2599/blog_demos.git	The warehouse address of the source code of the project, https protocol
git warehouse address (ssh)	[e-mail]:zq2599/blog_demos.git	The warehouse address of the source code of the project, ssh protocol

• There are multiple folders in this git project. The source code of this actual combat is under the <font color="blue">disruptor-tutorials</font> folder, as shown in the red box in the following figure:

• <font color="blue">disruptor-tutorials</font> is a parent project with multiple modules. The actual module in this article is <font color="red">low-level-operate</font>, as follows As shown in the red box:

Development

• Entering the coding stage, today’s task is to challenge the following three scenarios:
• 100 events, single consumer consumption;
• 100 events, three consumers, each consumes the 100 events alone;
• For 100 events, three consumers consume the 100 events together;
• Let's build the project first, then write the common code, such as event definition, event factory, etc., and finally the development of each scene;
• Add a module named <font color="red">low-level-operate</font> to the parent project <font color="blue">disruptor-tutorials</font>, and its build.gradle is as follows:

plugins {
    id 'org.springframework.boot'
}

dependencies {
    implementation 'org.projectlombok:lombok'
    implementation 'org.springframework.boot:spring-boot-starter'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'com.lmax:disruptor'

    testImplementation('org.springframework.boot:spring-boot-starter-test')
}

• Then the springboot startup class:

package com.bolingcavalry;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class LowLevelOperateApplication {
    public static void main(String[] args) {
        SpringApplication.run(LowLevelOperateApplication.class, args);
    }
}

• Event class, this is the definition of the event:

package com.bolingcavalry.service;

import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.ToString;

@Data
@ToString
@NoArgsConstructor
public class StringEvent {
    private String value;
}

• Event factory, which defines how to create event objects in memory:

package com.bolingcavalry.service;

import com.lmax.disruptor.EventFactory;

public class StringEventFactory implements EventFactory<StringEvent> {
    @Override
    public StringEvent newInstance() {
        return new StringEvent();
    }
}

• The event production class defines how to turn business logic events into disruptor events and publish them to the circular queue for consumption:

package com.bolingcavalry.service;

import com.lmax.disruptor.RingBuffer;

public class StringEventProducer {

    // 存储数据的环形队列
    private final RingBuffer<StringEvent> ringBuffer;

    public StringEventProducer(RingBuffer<StringEvent> ringBuffer) {
        this.ringBuffer = ringBuffer;
    }

    public void onData(String content) {

        // ringBuffer是个队列，其next方法返回的是下最后一条记录之后的位置，这是个可用位置
        long sequence = ringBuffer.next();

        try {
            // sequence位置取出的事件是空事件
            StringEvent stringEvent = ringBuffer.get(sequence);
            // 空事件添加业务信息
            stringEvent.setValue(content);
        } finally {
            // 发布
            ringBuffer.publish(sequence);
        }
    }
}

• Event processing class, specific business processing logic after receiving the event:

package com.bolingcavalry.service;

import com.lmax.disruptor.EventHandler;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import java.util.function.Consumer;

@Slf4j
public class StringEventHandler implements EventHandler<StringEvent> {

    public StringEventHandler(Consumer<?> consumer) {
        this.consumer = consumer;
    }

    // 外部可以传入Consumer实现类，每处理一条消息的时候，consumer的accept方法就会被执行一次
    private Consumer<?> consumer;

    @Override
    public void onEvent(StringEvent event, long sequence, boolean endOfBatch) throws Exception {
        log.info("sequence [{}], endOfBatch [{}], event : {}", sequence, endOfBatch, event);

        // 这里延时100ms，模拟消费事件的逻辑的耗时
        Thread.sleep(100);

        // 如果外部传入了consumer，就要执行一次accept方法
        if (null!=consumer) {
            consumer.accept(null);
        }
    }
}

• Define an interface, the external method of calling the interface to produce messages, and then put a few constants in it will be used later:

package com.bolingcavalry.service;

public interface LowLevelOperateService {
    /**
     * 消费者数量
     */
    int CONSUMER_NUM = 3;

    /**
     * 环形缓冲区大小
     */
    int BUFFER_SIZE = 16;

    /**
     * 发布一个事件
     * @param value
     * @return
     */
    void publish(String value);

    /**
     * 返回已经处理的任务总数
     * @return
     */
    long eventCount();
}

• The above is the public code, and then implement the three scenarios previously planned one by one;

• 100 events, single consumer consumption

• This is the simplest function. It implements the functions of publishing messages and single consumer consumption. The code is as follows. There are several points to note that will be mentioned later:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.*;
import com.lmax.disruptor.BatchEventProcessor;
import com.lmax.disruptor.RingBuffer;
import com.lmax.disruptor.SequenceBarrier;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;

import javax.annotation.PostConstruct;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;

@Service("oneConsumer")
@Slf4j
public class OneConsumerServiceImpl implements LowLevelOperateService {

    private RingBuffer<StringEvent> ringBuffer;

    private StringEventProducer producer;

    /**
     * 统计消息总数
     */
    private final AtomicLong eventCount = new AtomicLong();

    private ExecutorService executors;

    @PostConstruct
    private void init() {
        // 准备一个匿名类，传给disruptor的事件处理类，
        // 这样每次处理事件时，都会将已经处理事件的总数打印出来
        Consumer<?> eventCountPrinter = new Consumer<Object>() {
            @Override
            public void accept(Object o) {
                long count = eventCount.incrementAndGet();
                log.info("receive [{}] event", count);
            }
        };

        // 创建环形队列实例
        ringBuffer = RingBuffer.createSingleProducer(new StringEventFactory(), BUFFER_SIZE);

        // 准备线程池
        executors = Executors.newFixedThreadPool(1);

        //创建SequenceBarrier
        SequenceBarrier sequenceBarrier = ringBuffer.newBarrier();

        // 创建事件处理的工作类，里面执行StringEventHandler处理事件
        BatchEventProcessor<StringEvent> batchEventProcessor = new BatchEventProcessor<>(
                ringBuffer,
                sequenceBarrier,
                new StringEventHandler(eventCountPrinter));

        // 将消费者的sequence传给环形队列
        ringBuffer.addGatingSequences(batchEventProcessor.getSequence());

        // 在一个独立线程中取事件并消费
        executors.submit(batchEventProcessor);

        // 生产者
        producer = new StringEventProducer(ringBuffer);
    }

    @Override
    public void publish(String value) {
        producer.onData(value);
    }

    @Override
    public long eventCount() {
        return eventCount.get();
    }
}

• The above code has the following points to note:

1. Create your own RingBuffer instance of the ring queue
2. Prepare your own thread pool, the threads inside are used to obtain and consume messages
3. Do it yourself to create an instance of BatchEventProcessor and pass in the event processing class
4. Create a sequenceBarrier through ringBuffer and pass it to the BatchEventProcessor instance for use
5. Pass the sequence of BatchEventProcessor to ringBuffer to ensure that there will be no confusion in the production and consumption of ringBuffer
6. Starting the thread pool means that the BatchEventProcessor instance continuously obtains and consumes events from the ringBuffer in a separate thread;

• In order to verify whether the above code can work normally, I wrote a unit test class here, as shown below, the logic is very simple, call the OneConsumerServiceImpl.publish method one hundred times, generate one hundred events, and then check that the total number of consumption events recorded by OneConsumerServiceImpl is Is not equal to one hundred:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.LowLevelOperateService;
import lombok.extern.slf4j.Slf4j;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import static org.junit.Assert.assertEquals;

@RunWith(SpringRunner.class)
@SpringBootTest
@Slf4j
public class LowLeverOperateServiceImplTest {

    @Autowired
    @Qualifier("oneConsumer")
    LowLevelOperateService oneConsumer;

    private static final int EVENT_COUNT = 100;

    private void testLowLevelOperateService(LowLevelOperateService service, int eventCount, int expectEventCount) throws InterruptedException {
        for(int i=0;i<eventCount;i++) {
            log.info("publich {}", i);
            service.publish(String.valueOf(i));
        }

        // 异步消费，因此需要延时等待
        Thread.sleep(10000);

        // 消费的事件总数应该等于发布的事件数
        assertEquals(expectEventCount, service.eventCount());
    }

    @Test
    public void testOneConsumer() throws InterruptedException {
        log.info("start testOneConsumerService");
        testLowLevelOperateService(oneConsumer, EVENT_COUNT, EVENT_COUNT);
    }

• Note that if you are directly clicking the icon on IDEA to execute the unit test, remember to check the option in the red box in the figure below, otherwise the compilation may fail:

• Execute the above unit test class, the result is as shown in the following figure, the production and consumption of the message are in line with expectations, and the consumption logic is executed in a separate thread:

• Continue to challenge the next scene;

100 events, three consumers, each consumes the 100 events individually

• This scenario is also present in Kafka, that is, the groups of the three consumers are different, so that for each message, the two consumers consume each of them once;
• Therefore, for 100 events, each of 3 consumers will consume these 100 events independently, a total of 300 consumptions;
• The code is as follows, there are several points to be noted later:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.*;
import com.lmax.disruptor.BatchEventProcessor;
import com.lmax.disruptor.RingBuffer;
import com.lmax.disruptor.SequenceBarrier;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;

import javax.annotation.PostConstruct;
import java.util.concurrent.Executor;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;

@Service("multiConsumer")
@Slf4j
public class MultiConsumerServiceImpl implements LowLevelOperateService {

    private RingBuffer<StringEvent> ringBuffer;

    private StringEventProducer producer;

    /**
     * 统计消息总数
     */
    private final AtomicLong eventCount = new AtomicLong();

    /**
     * 生产一个BatchEventProcessor实例，并且启动独立线程开始获取和消费消息
     * @param executorService
     */
    private void addProcessor(ExecutorService executorService) {
        // 准备一个匿名类，传给disruptor的事件处理类，
        // 这样每次处理事件时，都会将已经处理事件的总数打印出来
        Consumer<?> eventCountPrinter = new Consumer<Object>() {
            @Override
            public void accept(Object o) {
                long count = eventCount.incrementAndGet();
                log.info("receive [{}] event", count);
            }
        };

        BatchEventProcessor<StringEvent> batchEventProcessor = new BatchEventProcessor<>(
                ringBuffer,
                ringBuffer.newBarrier(),
                new StringEventHandler(eventCountPrinter));

        // 将当前消费者的sequence实例传给ringBuffer
        ringBuffer.addGatingSequences(batchEventProcessor.getSequence());

        // 启动独立线程获取和消费事件
        executorService.submit(batchEventProcessor);
    }

    @PostConstruct
    private void init() {
        ringBuffer = RingBuffer.createSingleProducer(new StringEventFactory(), BUFFER_SIZE);

        ExecutorService executorService = Executors.newFixedThreadPool(CONSUMER_NUM);

        // 创建多个消费者，并在独立线程中获取和消费事件
        for (int i=0;i<CONSUMER_NUM;i++) {
            addProcessor(executorService);
        }

        // 生产者
        producer = new StringEventProducer(ringBuffer);
    }

    @Override
    public void publish(String value) {
        producer.onData(value);
    }

    @Override
    public long eventCount() {
        return eventCount.get();
    }
}

• The above code is not much different from the previous OneConsumerServiceImpl, mainly because multiple BatchEventProcessor instances are created and then submitted in the thread pool respectively;
• The verification method is still a unit test. Just add the code in LowLeverOperateServiceImplTest.java. Note that the third parameter of testLowLevelOperateService is <font color="blue">EVENT_COUNT * LowLevelOperateService.CONSUMER_NUM</font>, which means it is expected to be consumed The number of messages is <font color="red"> 300 </font>:

     @Autowired
    @Qualifier("multiConsumer")
    LowLevelOperateService multiConsumer;

    @Test
    public void testMultiConsumer() throws InterruptedException {
        log.info("start testMultiConsumer");
        testLowLevelOperateService(multiConsumer, EVENT_COUNT, EVENT_COUNT * LowLevelOperateService.CONSUMER_NUM);
    }

• Perform unit test, as shown in the following figure, a total of 300 events are consumed, and three consumers are not moving threads:

100 events, three consumers consume the 100 events together

• The last actual battle in this article is to publish 100 events, and then let three consumers consume 100 together (for example, A consumes 33, B consumes 33, and C consumes 34);
• The BatchEventProcessor used earlier is used for independent consumption and is not suitable for multiple consumers to consume together. This scenario of multiple consumption and common consumption needs to be completed with the help of WorkerPool. The name is still very vivid: there are many jobs in a pool If you put the task into this pool, each of the workers will handle part of it, and everyone will work together to complete the task;
• Consumers passing in WorkerPool need to implement the WorkHandler interface, so a new implementation class is added:

package com.bolingcavalry.service;

import com.lmax.disruptor.WorkHandler;
import lombok.extern.slf4j.Slf4j;
import java.util.function.Consumer;

@Slf4j
public class StringWorkHandler implements WorkHandler<StringEvent> {

    public StringWorkHandler(Consumer<?> consumer) {
        this.consumer = consumer;
    }

    // 外部可以传入Consumer实现类，每处理一条消息的时候，consumer的accept方法就会被执行一次
    private Consumer<?> consumer;

    @Override
    public void onEvent(StringEvent event) throws Exception {
        log.info("work handler event : {}", event);

        // 这里延时100ms，模拟消费事件的逻辑的耗时
        Thread.sleep(100);

        // 如果外部传入了consumer，就要执行一次accept方法
        if (null!=consumer) {
            consumer.accept(null);
        }
    }
}

• New service classes are added to realize the logic of common consumption. There are several points to note that will be mentioned later:

package com.bolingcavalry.service.impl;

import com.bolingcavalry.service.*;
import com.lmax.disruptor.*;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Consumer;

@Service("workerPoolConsumer")
@Slf4j
public class WorkerPoolConsumerServiceImpl implements LowLevelOperateService {

    private RingBuffer<StringEvent> ringBuffer;

    private StringEventProducer producer;

    /**
     * 统计消息总数
     */
    private final AtomicLong eventCount = new AtomicLong();

    @PostConstruct
    private void init() {
        ringBuffer = RingBuffer.createSingleProducer(new StringEventFactory(), BUFFER_SIZE);

        ExecutorService executorService = Executors.newFixedThreadPool(CONSUMER_NUM);

        StringWorkHandler[] handlers = new StringWorkHandler[CONSUMER_NUM];

        // 创建多个StringWorkHandler实例，放入一个数组中
        for (int i=0;i < CONSUMER_NUM;i++) {
            handlers[i] = new StringWorkHandler(o -> {
                long count = eventCount.incrementAndGet();
                log.info("receive [{}] event", count);
            });
        }

        // 创建WorkerPool实例，将StringWorkHandler实例的数组传进去，代表共同消费者的数量
        WorkerPool<StringEvent> workerPool = new WorkerPool<>(ringBuffer, ringBuffer.newBarrier(), new IgnoreExceptionHandler(), handlers);

        // 这一句很重要，去掉就会出现重复消费同一个事件的问题
        ringBuffer.addGatingSequences(workerPool.getWorkerSequences());

        workerPool.start(executorService);

        // 生产者
        producer = new StringEventProducer(ringBuffer);
    }

    @Override
    public void publish(String value) {
        producer.onData(value);
    }

    @Override
    public long eventCount() {
        return eventCount.get();
    }
}

• In the above code, there are two things to pay attention to:

1. After the StringWorkHandler array is passed to WorkerPool, each StringWorkHandler instance is put into a new WorkProcessor instance. WorkProcessor implements the Runnable interface. When executing <font color="blue">workerPool.start</font>, WorkProcessor will be submitted To the thread pool;
2. Compared with the previous independent consumption, the biggest feature of joint consumption is that it only calls the <font color="blue">ringBuffer.addGatingSequences</font> method once, which means that three consumers share a sequence instance;
3. The verification method is still a unit test. Just add the code in LowLeverOperateServiceImplTest.java. Note that the third parameter of testWorkerPoolConsumer is <font color="blue">EVENT_COUNT</font>, which means that the expected number of messages consumed is < font color="red"> 100 </font>:

     @Autowired
    @Qualifier("workerPoolConsumer")
    LowLevelOperateService workerPoolConsumer;
    
    @Test
    public void testWorkerPoolConsumer() throws InterruptedException {
        log.info("start testWorkerPoolConsumer");
        testLowLevelOperateService(workerPoolConsumer, EVENT_COUNT, EVENT_COUNT);
    }

• The execution unit test is shown in the following figure. Three consumers consume a total of 100 events, and the three consumers are in different threads:

• So far, we have completed the message production and consumption of three common scenarios without using the Disruptor class. I believe you also have a deep understanding of the underlying implementation of the Disruptor. In the future, whether you use or optimize the Disruptor, you will definitely be more comfortable;

You are not lonely, Xinchen is with you all the way

1. Java series
2. Spring series
3. Docker series
4. kubernetes series
5. database + middleware series
6. DevOps series

Welcome to pay attention to the public account: programmer Xin Chen

Search "Programmer Xin Chen" on WeChat, I am Xin Chen, and I look forward to traveling the Java world with you...
https://github.com/zq2599/blog_demos