JAVA language asynchronous non-blocking design pattern (application articles)

首图.gif

1 Overview

There are 2 articles in this series. In the last article "Principles", we saw the asynchronous non-blocking model, which can effectively reduce the time consumption of thread IO state, improve resource utilization and system throughput. Asynchronous API can be expressed in the form of listener or Promise; the Promise API provides greater flexibility, supports synchronous returns and asynchronous callbacks, and allows any number of callbacks to be registered.

In this article "Application", we will further explore the application of asynchronous mode and Promise:

Chapter 2: Promise and thread pool. executes time-consuming requests asynchronously, ExecutorService+Future is an alternative; but compared to Future, Promise supports pure asynchronous acquisition of response data, which can eliminate more congestion.

Chapter 3: Exception Handling. Java programs do not always successfully execute requests, and sometimes encounter force majeure such as network problems. For unavoidable abnormal situations, the asynchronous API must provide an exception handling mechanism to improve the fault tolerance of the program.

Chapter 4: Request Scheduling. Java programs sometimes need to submit multiple requests, and there may be a certain correlation between these requests, including sequential execution, parallel execution, and batch execution. Asynchronous APIs need to provide support for these constraints.

This article does not limit the specific implementation of Promise. Readers can choose a Promise tool class in the production environment (such as netty DefaultPromise[A], jdk CompletableFuture[B], etc.); in addition, since the principle of Promise is not complicated, readers can also implement it by themselves. Need function.

2.Promise and thread pool

Java programs sometimes need to perform time-consuming IO operations, such as database access; during this period, compared to pure memory computing, the duration of IO operations is significantly longer. In order to reduce IO congestion and improve resource utilization, we should use the asynchronous model to submit requests to other threads for execution, so as to submit multiple requests continuously without waiting for the previous request to return.

This chapter compares several IO models (see section 2.1) and examines the blocking of the caller's thread. Among them, Promise supports purely asynchronous request submission and response data processing, which can eliminate unnecessary blocking to the greatest extent. In actual projects, if the underlying API does not support pure asynchrony, then we can also refactor appropriately to make it compatible with Promise (see section 2.2).

2.1 Comparison: Synchronization, Future, Promise

This section compares several IO models, including synchronous IO, thread pool (ExecutorService)-based asynchronous IO, and Promise-based asynchronous IO, and examines the blocking of the caller's thread. Suppose we want to execute a database access request. Due to the need to cross the network, a single request requires time-consuming IO operations to finally receive the response data; but there are no constraints between the requests, allowing new requests to be submitted at any time without receiving the previous response data.

First we look at sample code for several models of :

1. Synchronous IO . The db.writeSync() method is synchronously blocking. The function blocks until the response data is received. Therefore, the caller can only submit one request at a time, and must wait for the request to return before submitting the next request.

/* 提交请求并阻塞，直至收到响应数据*/
String result = db.writeSync("data");
process(result);

based on the thread pool (ExecutorService). The db.writeSync() method remains unchanged; but submit it to the thread pool for execution, so that the caller thread will not be blocked, so that multiple requests data1-3 can be submitted continuously.

After submitting the request, the thread pool returns the Future object, and the caller calls Future.get() to obtain the response data. The Future.get() method is blocking, so the caller cannot submit subsequent requests until the response data is obtained.

/* 提交请求*/
// executor: ExecutorService
Future<String> resultFuture1 = executor.submit(() -> db.writeSync("data1"));
Future<String> resultFuture2 = executor.submit(() -> db.writeSync("data2"));
Future<String> resultFuture3 = executor.submit(() -> db.writeSync("data3"));

/* 获取响应：同步*/
String result1 = resultFuture1.get();
String result2 = resultFuture2.get();
String result3 = resultFuture3.get();
process(result1);
process(result2);
process(result3);

3. Promise-based asynchronous IO . The db.writeAsync() method is purely asynchronous and returns a Promise object after submitting the request; the caller calls Promise.await() to register the callback, and the callback is triggered when the response data is received.

In "Principles" , we saw that the Promise API can be implemented based on a thread pool or a responsive model; either way, the callback function can be executed in the thread that receives the response, without the caller thread blocking waiting for the response data.

/* 提交请求*/
Promise<String> resultPromise1 = db.writeAsync("data1");
Promise<String> resultPromise2 = db.writeAsync("data2");
Promise<String> resultPromise3 = db.writeAsync("data3");

/* 获取响应：异步*/
resultPromise1.await(result1 -> process(result1));
resultPromise2.await(result2 -> process(result2));
resultPromise3.await(result3 -> process(result3));

Next, let's look at the process of the caller thread state changing over time in the above models, as shown in Figure 2-1.

a. synchronous IO . The caller can only submit one request at a time, and cannot submit the next request until the response is received.
b. based on thread pool. The same group of requests (requests 1-3, and requests 4-6) can be submitted consecutively without waiting for the return of the previous request. However, once the caller uses Future.get() to get the response data (result1-3), it will block and cannot submit the next set of requests (requests 4-6) until the response data is actually received.
c. Promise-based asynchronous IO. caller can submit a request at any time and register a callback function for the response data with the Promise; later the receiving thread notifies the Promise of the response data to trigger the callback function. In the above process, the caller thread does not need to wait for the response data and will never block.

Figure 2-1a Thread timeline: Synchronous IO

Figure 2-1b Thread timeline: Asynchronous IO based on thread pool

Figure 2-1c Thread timeline: Promise-based asynchronous IO

2.2 Promise combined with thread pool

Compared with ExecutorService+Future, Promise has the advantage of pure asynchronous; however, in some scenarios, it is also necessary to use Promise and thread pool in combination. For example: 1. The underlying API only supports the synchronous blocking model, and does not support pure asynchronous; at this time, the API can only be called in the thread pool to achieve non-blocking. 2. A piece of legacy code needs to be refactored to change its thread model from a thread pool model to a responsive model; you can first change the external interface to Promise API, and the underlying implementation temporarily uses the thread pool.

The following code snippet shows the Promise combined with thread pool:

Create a Promise object as the return value. Note that PromiseOrException is used here, in case an exception is encountered during the period; it can notify the response data, or notify the thrown Exception when it fails. See section 3.1 for details.
Execute the request in the thread pool (2a), and notify the Promise after receiving the response data (2b)
Handle the thread pool full exception. The bottom layer of the thread pool is associated with a BlockingQueue to store the tasks to be executed. It is generally set as a bounded queue to prevent unlimited memory usage. When the queue is full, a task will be discarded. In order to notify the caller of the exception, the rejection policy of the thread pool must be set to AbortPolicy. When the queue is full, the submitted task will be discarded and RejectedExecutionException will be thrown; once the exception is caught, the Promise must be notified that the request has failed.

public PromiseOrException<String, Exception> writeAsync() {
// 1.创建Promise对象
    PromiseOrException<String, Exception> resultPromise = new PromiseOrException<>();
    try {
        executor.execute(() -> {
            String result = db.writeSync("data"); // 2a.执行请求。只支持同步阻塞
            resultPromise.signalAllWithResult(result); // 2b.通知Promise
        });

    }catch (RejectedExecutionException e){ // 3.异常：线程池满
        resultPromise.signalAllWithException(e); 
    }

    return resultPromise;
}

3. Exception handling: PromiseOrException

Java programs sometimes encounter inevitable abnormal situations, such as network disconnection; therefore, programmers need to design appropriate exception handling mechanisms to improve the program's fault tolerance. This chapter introduces the exception handling of asynchronous API, first introduces the Java language exception handling specification; then introduces PromiseOrException, a variant of Promise, so that the Promise API supports standardized exception handling.

3.1 Exception handling specifications

I personally believe that the exception handling Java code should comply with the following specifications:

Explicitly distinguish between normal exits and abnormal exits.
Supports compile-time checking, forcing callers to deal with inevitable exceptions.

distinguish between normal exits and abnormal exits

Exceptions are an important feature of the Java language and a basic control flow. In the Java language, a function is allowed to have one return value and throw multiple different types of exceptions. The return value of the function is a normal exit. The function return indicates that the function can work normally and calculates the correct result; on the contrary, once the function encounters an abnormal situation and cannot continue to work, such as network disconnection, illegal request, etc., it must throw the corresponding The exception.

Although if-else and exception are both control flow, the programmer must between . The branches of if-else are generally equivalent and are used to handle normal situations; while the return value and exception of the function are not equal, throwing an exception means that the function has encountered an unhandled fault and cannot calculate the result normally. It is fundamentally different from the return value generated by the normal function of the function. In API design, confusing normal exits (return values) and exception exits (throwing exceptions), or not throwing exceptions when you cannot continue working, are serious design flaws.

Taking database access as an example, the following code compares the two forms of API exception handling. During database access, if the network connection is smooth and the server can process the request correctly, then db.write() should return the response data of the server, such as the self-increment id generated by the server for the data written, and the data actually affected by the conditional update The number of entries, etc.; if the network connection is disconnected, or the client and server versions do not match, the request cannot be parsed and cannot work normally, then db.write() should throw an exception to explain the specific reason. From the perspective of "whether it works normally", the nature of the above two cases is completely different. Obviously, exceptions should be used as control flow instead of if-else.

/* 正确*/
try {
    String result = db.write("data");
    process(result); // 正常出口
} catch (Exception e) {
    log.error("write fails", e); // 异常出口
}

/* 错误*/
String resultOrError = db.write("data");
if (resultOrError.equals("OK")) {
    process(resultOrError); // 正常出口
} else {
    log.error("write fails, error: " + resultOrError); // 异常出口
}

Mandatory handling of inevitable exceptions

In the exception handling system of the Java language, exceptions are mainly divided into the following categories : Exception, RuntimeException, Error; the three are all subclasses of Throwable, that is, they can be thrown by functions. Note that because RuntimeException is a subclass of Exception, in this article, to avoid confusion, "Exception" specifically refers to those exceptions that are "Exception but not RuntimeException".

In my opinion, are used in the following scenarios :

1. Exception : Abnormal conditions caused by force majeure outside the program, such as network disconnection. Even if the Java code is flawless, it is absolutely impossible to avoid such anomalies (unplug the network cable and try!). Since it is unavoidable, this kind of exception should be handled forcibly to improve the fault tolerance of the system.
2. RuntimeException : abnormal conditions caused by programming errors, such as the array subscript out of bounds ArrayOutOfBoundException, the parameter does not meet the value range IllegalArgumentException, etc. If the programmer is well aware of the API input constraints, and performs proper verification of the function parameters before calling the API, then RuntimeException can be absolutely avoided (unless the called API actually throws a RuntimeException where it should throw an Exception). Since it can be avoided, there is no need to force handling of this exception.
Of course, no one is perfect. Assuming that the programmer really violates certain constraints, the function throws RuntimeException and is not processed, then as a punishment, the thread or process will exit, thereby reminding the programmer to correct the error code. If the thread or process must be resident, it is necessary to cover the RuntimeException, as shown in the following code. Here, the code defect is regarded as an unavoidable abnormal situation. After the abnormality is caught, the log can be recorded, an alarm can be triggered, and the defect can be corrected later.
new Thread(()->{
   while (true){
       try{
           doSomething();
       }catch (RuntimeException e){  // 对RuntimeException进行兜底，以防线程中断
 log.error("error occurs", e);
       }
   }
});
3. Error: the exception defined inside the jvm, such as OutOfMemoryError. Business logic generally does not throw Error, but throws some kind of Exception or RuntimeException.

Among the above-mentioned types of exceptions, only Exception is mandatory, which is called checked exception[C]. The following is an example of a checked exception. Database access DB.write() throws Exception, indicating that it encounters a force majeure situation such as network disconnection and message parsing failure. The exception type is Exception instead of RuntimeException, in order to force the caller to add a catch clause to handle the above situation; if the caller misses the catch clause, the compiler will report an error, prompting the caller, "There must be an exception here, and it must be done Processing" to improve the fault tolerance of the program.

 /**
 * 抛出异常，如果：
 * 1.网络连接断开
 * 2. 消息无法解析
 * 3. 业务逻辑相关，如服务端扣款时发现余额不足
 * 4. …… // 任何无法避免的情况，都应该抛出Exception！
 */
 public String write(Object data) throws Exception {
    return "";
}

 /**
 * 处理异常
 */
try {
    String result = db.write("data");
    process(result); 
 } catch (Exception e) {  // 如遗漏catch子句，则编译不通过
    log.error("write fails, db: ..., data: ...", e);
 }

3.2 Exception handling of Promise API

The previous section discussed the specification of exception handling:

Explicitly distinguish between normal exits and abnormal exits;
Irresistible exceptions must be forcibly handled at compile time. The following code shows how the Promise API should design an exception handling mechanism to comply with the above specifications.

uses PromiseOrException to notify response data and exception . PromiseOrException<T, E> is a subclass of Promise<X>, the generic template X is the data object ResultOrException<T, E extends Exception>, which contains two fields result and e: e==null means normal. At this time, the field result is valid; e!=null indicates an exception, do not use the field result at this time.
In "Reload 1", the caller obtains the ResultOrException object from the callback function. Call ResultOrException.get() to get the response data result, or get() method throws exception e. The code structure of this method is consistent with traditional exception handling, and multiple catch clauses can be used to handle different types of exceptions.
In "Overload 2", the caller obtains result and e directly from the callback function. The meaning is the same as above. This way saves ResultOrException.get(); but if you need to deal with different types of exceptions, you need to use e instanceof MyException to determine the type of exception.

// extends Promise<ResultOrException<String, Exception>>
 PromiseOrException<String, Exception> resultPromsie = db.writeAsync("data");

 /* 重载1*/ 
resultPromsie.await(resultOrException -> { 
    try {
        String result = resultOrException.get();
        process(result);  // 正常出口
     } catch (Exception e) {
        log.error("write fails", e);  // 异常出口
 }
});
 
 /* 重载2*/
 resultPromsie.await((result, e) -> {
    if (e == null) {
        process(result);  // 正常出口
 } else {
        log.error("write fails", e);  // 异常出口
 }
});

PromiseOrException conforms to the exception handling specifications proposed in the previous section, has the following advantages :

Distinguish between normal exits and abnormal exits. Response data and exceptions are passed using the result and e variables respectively, and e==null can be used to judge whether it is normal. Note that result==null cannot be used as a judgment condition, because null may be a legal value of the response data.
Force exceptions. No matter which kind of callback is used, there is no code structure that can only obtain result without obtaining e, so the exception handling of e is not omitted in the syntax.
Allows the definition of exception types. It is not necessary for the generic template E of PromiseOrException to be filled with Excetion, and it can be filled with any other type. Note that due to Java grammar, only one type of exception is allowed to be filled in the generic template, instead of allowing multiple exceptions to be thrown like functions throwing exceptions. To cope with this limitation, we can only define an exception parent class for the API, and the caller uses the catch clause or instanceof to perform down-casting. Of course, this method of "defining the exception parent class" is also acceptable and widely used in existing tools, because the exception thrown by the tool can be distinguished from the built-in exception type of the Java language.

Finally, the proposed structure of exception handling personal a suggestion : All anomalies notified by PromiseOrException, while the API itself never throw an exception. Taking the database access API writeAsync() as an example, the code above compares two ways of throwing exceptions. correct approach is PromiseOrException as the only exit. If the underlying implementation of the API throws an exception (submit() throws Exception), the exception should be encapsulated in the PromiseOrException object instead of directly thrown from the API function (writeAsync() throws Exception) ).

/* 正确：唯一出口PromiseOrException*/
public PromiseOrException<String, Exception> writeAsync(Object data)  {
    try {
        submit(data);  // throws exception
    } catch (Exception e) {
        return PromiseOrException.immediatelyException(e);
    }
    PromiseOrException<String, Exception> resultPromise = new PromiseOrException<>();
    return resultPromise;
}

/* 错误：两个出口throws Exception和PromiseOrException*/
public PromiseOrException<String, Exception> writeAsync(Object data) throws Exception {
    submit(data);  // throws exception

    PromiseOrException<String, Exception> resultPromise = new PromiseOrException<>();
    return resultPromise;
}

If an API with two exception exits is wrongly designed, the caller will have to write the exception handling logic repeatedly, as shown in the following code.

try {
    PromiseOrException<String, Exception> resultPromise = db.writeAsync("data");
    resultPromise.await((result, e) -> {
        if (e == null) {
            process(result);  // 正常出口
 } else {
            log.error("write fails", e);  // 异常出口2
 }
    });
} catch (Exception e) {
    log.error("write fails", e);  // 异常出口1
}

4. Request scheduling

In Java programs, it is sometimes necessary to submit multiple asynchronous requests, and there is a certain correlation between these requests. In the asynchronous non-blocking scenario, these relationships can all be realized with the help of Promises.

1. Sequence request , as shown in Figure 4-1. The next request depends on the response data of the previous request; therefore, you must wait for the previous request to return before constructing and submitting the next request.

Figure 4-1 Sequence request

2. Parallel request , as shown in Figure 4-2. Submit multiple requests at once, and then wait for all requests to return. There is no dependency between the submitted requests, so they can be executed at the same time; but the response data of each request must be received (the channelRead() event occurs, and the event parameter is the response data) before the actual processing process(result1,2, 3).

Figure 4-2 Parallel request

3. Batch request , as shown in Figure 4-3. The caller submits multiple requests in a row, but they are temporarily stored in the queue (offer()) instead of being executed immediately. After a period of time, several requests are taken out of the queue and assembled into batch requests for submission (writeBulk()); when the response message of the batch request is received, the response data of each request can be taken out. Since each network IO brings additional overhead, batch requests are often used in practical applications to reduce the frequency of network IO to improve overall throughput.

Figure 4-3 Batch request

4.1. Sequential request: Promise.then()

Assuming that a series of operations need to be completed in sequence, that is, after the previous operation is completed, the next operation can be executed; if these operations are all represented as Promise API, we can encapsulate Promise.await(listener) to make the code structure more concise.

The following is an asynchronous Promise API. The submit method submits the request and returns a Promise object; when the response data is received, the Promise object is notified.

/**
 * 异步Promise API
 */
 public static Promise<String> submit(Object request) {
    Promise<String> resultPromise = new Promise<>();
     // ……
 return resultPromise;
}

Now suppose there are 5 requests called "A"-"E", and these requests must be submitted in turn. For example, because the parameters of request B depend on the response data of request A, after submitting request A, the response data resultA must be processed before request B can be submitted. This scenario can be implemented with the code shown below. After a certain call to the submit("X") function, we register a callback on the returned Promise object; the response data resultX is processed in the callback function, and submit("X+1") is called to submit the next request.

Although this approach can achieve functional requirements, the nested code structure is very readable-each additional request requires more nesting and indentation of the code by one level. When the calling logic is complex and the number of requests is large, the code will be very difficult to maintain.

This situation is also called "callback hell" [D], there are many related discussions in the JavaScript language, which can be used as a reference.

submit("A").await(resultA -> {
    submit("B").await(resultB -> {
        submit("C").await(resultC -> {
            submit("D").await(resultD -> {
                submit("E").await(resultE -> {
                    process(resultE);
                });
            });
        });
    });
});

To improve the code structure, we encapsulate the Promise<T>.await(Consumer<T>) method and provide the Promise<T>.then(Function<T, Promise<Next>>) method, as shown below. Similar to await(), then() can also register a callback function resultX->submit("X+1"), the callback function processes the response data resultX, and submits the next request submit("X+1"); then( The return value of) is the return value of submit("X+1"), which is used to notify the response data resultX+1 of the next request.

Promise<String> resultPromiseA = submit("A");
Promise<String> resultPromiseB = resultPromiseA.then(resultA -> submit("B"));
Promise<String> resultPromiseC = resultPromiseB.then(resultB -> submit("C"));
Promise<String> resultPromiseD = resultPromiseC.then(resultC -> submit("D"));
Promise<String> resultPromiseE = resultPromiseD.then(resultD -> submit("E"));
resultPromiseE.await(resultE -> process(resultE));

Next, we inline the intermediate variable resultPromiseA-E to get a chain call structure based on then(). Compared to await(), then() eliminates nested callbacks like a doll.

submit("A")
        .then(resultA -> submit("B")) // 返回resultPromiseB
        .then(resultB -> submit("C")) // 返回resultPromiseC
        .then(resultC -> submit("D")) // 返回resultPromiseD
        .then(resultD -> submit("E")) // 返回resultPromiseE
        .await(resultE -> process(resultE));

Finally, we look at Promise <T> one kind .then () is a simple implementation , as follows:

The then() method provides a generic template Next to indicate the response data type of the next request.
According to the generic template Next, then() internally creates Promise<Next> as the return value to notify the response data of the next request.
For the current request, call await() to register the callback result of the response data; when the response data is received, the function func is executed to submit the next request: func.apply(result).
When the response data of the next request is received, Promise<Next> is notified: nextPromise::signalAll.

public <Next> Promise<Next> then(Function<T, Promise<Next>> func) {
    Promise<Next> nextPromise = new Promise<>();
    await(result -> {
        Promise<Next> resultPromiseNext = func.apply(result);
        resultPromiseNext.await(nextPromise::signalAll);
    });
    return nextPromise;
}

Note that only the pure asynchronous overload of Promise<T>.then(Function<T, Promise<Next>>) is shown here. Depending on whether the callback function has a return value, is executed synchronously or asynchronously, Promise can provide more overloads of then(); limited by Java syntax, if the compiler cannot distinguish each overload, you can use the function name to make it explicit The difference, such as:

thenRun(Runnable)

thenAccept(Consumer<T>)

thenApply(Function<T, Next>)

thenApplyAsync(Function<T, Promise<Next>>)

4.2. Parallel request: LatchPromise

The previous section introduced the "sequential request" scenario, that is, multiple requests need to be executed in sequence; while in the "parallel request" scenario, there is no order constraint between multiple requests, but we still need to wait for all requests to return before performing subsequent operations . For example, we need to query multiple database tables, these query statements can be executed at the same time; but must wait for each query to return, we can get complete information. jdk provides CountDownLatch to implement this scenario, but it only supports synchronous waiting; as an improvement, we use LatchPromise to achieve the same function and support pure asynchronous API.

Taking database access as an example, the code shown below shows the use of :

Submit 3 requests and get the Promise object resultPromise1-3 corresponding to each request to get the response data.
Create a LatchPromise object and register with it the Promise object resultPromise1-3 that needs to be waited for.
LatchPromise.untilAllSignaled() returns a Promise object allSignaled. When all registered resultPromise1-3 are notified, allSignaled will be notified. The type of allSignaled is VoidPromise, which means that there is no response data to be processed when allSignaled is notified.
Register the callback on allSignaled, call resultPromiseX.await() in the callback function to get the actual response data; at this time, since the request has been executed, await() returns immediately without blocking.

/* 创建Promise对象*/
Promise<String> resultPromise1 = db.writeAsync("a");
Promise<String> resultPromise2 = db.writeAsync("b");
Promise<String> resultPromise3 = db.writeAsync("c");

/* 向LatchPromise注册要等待的Promise*/
LatchPromise latch = new LatchPromise();
latch.add(resultPromise1);
latch.add(resultPromise2);
latch.add(resultPromise3);

/* 等待全部Promise被通知*/
VoidPromise allSignaled = latch.untilAllSignaled();
allSignaled.await(() -> {
    String result1 = resultPromise1.await();
    String result2 = resultPromise2.await();
    String result3 = resultPromise3.await();
    process(result1, result2, result3);
});

For comparison, the following code uses CountDownLatch to achieve the same function, but has the following defects :

CountDownLatch.await() only supports synchronous waiting. It is unacceptable in a purely asynchronous scenario.
CountDownLatch is invasive to business logic. Programmers need to add a call to CountDownLatch.countDown() in the business logic to control the timing of CountDownLatch; on the contrary, LatchPromise relies on the already existing resultPromise object without writing additional timing control code.
CountDownLatch introduces redundant logic. When creating CountDownLatch, you must fill in the number of requests to wait in the construction parameters; therefore, once the number of submitted requests changes, you must update the code for creating CountDownLatch and modify the construction parameters accordingly.

CountDownLatch latch = new CountDownLatch(3);
resultPromise1.await(result1 -> latch.countDown());
resultPromise2.await(result2 -> latch.countDown());
resultPromise3.await(result3 -> latch.countDown());

latch.await();
String result1 = resultPromise1.await();
String result2 = resultPromise2.await();
String result3 = resultPromise3.await();
process(result1, result2, result3);

Finally, we look at reference LatchPromise implementation . The code principle is as follows:

set up countUnfinished variable, the number of Promise object record has not been notified of . Every time a Promise object is registered, countUnfinished is incremented; every time a Promise is notified, countUnfinished is decremented. When countUnfinished is reduced to 0, it means that all registered Promise objects have been notified, so allSignaled is notified.
sets up the noMore variable to record whether new Promise objects need to be registered. Only when untilAllSignaled() is called is the registration completed; before that, even if countUnfinished is reduced to 0, allSignaled should not be notified. Consider a situation where you need to register and wait for resultPromise1-3, in which resultPromise1 and 2 have been notified during the registration period, and resultPromise3 has not been notified. If noMore is not judged, then after registering resultPromise1 and 2, countUnfinished has been reduced to 0, resulting in advance notification of allSignaled; this is a timing error, because in fact resultPromise3 has not yet been completed.
must be locked when accessing variables. Synchronized is used here.
Pay attention to . When calling untilAllSignaled(), if the initial value of countUnfinished is already 0, allSignaled should be notified immediately; because countUnfinished can no longer be decremented, there will be no chance to notify allSignaled afterwards.

// private static class Lock。无成员，仅用于synchronized(lock)
private final Lock lock = new Lock();
private int countUnfinished = 0;
private final VoidPromise allSignaled = new VoidPromise();

public void add(Promise<?> promise) {
    if (promise.isSignaled()) {
        return;
    }

    synchronized (lock) {
        countUnfinished++;
    }

    promise.await(unused -> {
        synchronized (lock) {
            countUnfinished--;
            if (countUnfinished == 0 && noMore) {
                allSignaled.signalAll();
            }
        }
    });
}

public VoidPromise untilAllSignaled() {
    synchronized (lock) {
        if (countUnfinished == 0) {
            allSignaled.signalAll();
        } else {
            noMore = true;
        }
    }

    return allSignaled;
}

4.3. Batch request: ExecutorAsync

Bulk request characteristics

"Batch request" (also known as "bulk" or "batch") means that multiple requests can be carried in one message, which is mainly used in scenarios such as database access and remote invocation. Since the number of network IOs is reduced, and the overhead of constructing and transmitting messages is saved, batch requests can effectively improve throughput.

Many database APIs support batch read and write, such as JDBC PreparedStatement[E], elasticsearch bulk API[F], mongo DB insertMany()[G], influx DB BatchPoints[H], readers can refer to the references for further understanding. In order to improve performance, some APIs sacrifice ease of use. Among them, the elasticsearch bulk API has the least restrictions on callers, allowing different types of requests such as mixed additions, deletions and modifications, and allows writing to different database tables (index); mongo DB and influx DB are the second, and a batch request can only be written to the same one. Database tables, but you can customize the fields of each data; PreparedStatement has the lowest flexibility, it defines the template of the SQL statement, the caller can only fill in the template parameters, but not modify the statement structure.

Although database APIs already support batch access, many native APIs still require callers to construct batch requests themselves, and callers need to handle complex details such as request assembly, batch size, and number of concurrent requests.

Here, we design the general component ExecutorAsync to encapsulate the request scheduling strategy to provide a more concise API. The usage flow of shown in the following code snippet:

Similar to the thread pool ExecutorService.submit(), the caller can call ExecutorAsync.submit() to submit a request. Among them, the request is represented by the data object Request, which is used to store the request type and request parameters.
After submitting the request, the caller obtains the Promise object to obtain the response data. Due to the use of Promises, ExecutorAsync supports purely asynchronous operations, and there is no need to block when submitting requests and obtaining response data.
ExecutorAsync internally schedules requests. Instead of submitting a request and executing it immediately, it collects a batch of requests at regular intervals, assembles them into a batch request, and then calls the actual database access API. If the database access API allows, then a batch of requests can be mixed with different request types, or operate on different database tables.

ExecutorAsync executor = new ExecutorAsync();
Promise<...> resultPromise1 = executor.submit(new Request("data1"));
Promise<...> resultPromise2 = executor.submit(new Request("data2"));
Promise<...> resultPromise3 = executor.submit(new Request("data3"));

Specifically, ExecutorAsync supports the following scheduling strategies:

1. Queue, as shown in Figure 4-4a. After the caller submits the request, do not execute it immediately, but cache it in the queue.

Figure 4-4a ExecutorAsync feature: queuing

2. Batch , as shown in Figure 4-4b. At regular intervals, ExecutorAsync takes out several requests from the queue, assembles them into bulk requests, and calls the underlying database API to submit them to the server. If the queue length grows quickly, we can also define a bulk size, and when the queue length reaches this value, a bulk request will be assembled and submitted immediately.

Figure 4-4b ExecutorAsync feature: batch

3. Concurrent , as shown in Figure 4-4c. If the underlying database API supports asynchronous submission of requests, then ExecutorAsync can make full use of this feature to submit multiple batch requests continuously without waiting for the return of the previous batch requests. To avoid overloading the database server, we can define parallelism to limit the number of batch requests being executed (in flight); when the limit is reached, if the caller submits a new request, it will be temporarily stored in the queue waiting for execution, and New batch requests will not be assembled.

Figure 4-4c ExecutorAsync feature: concurrent

4. Discard . As shown in Figure 4-4d. Under the restrictions of bulk size and parallelism mentioned above, if the rate of submitting requests is much higher than the rate of server response, then a large number of requests will accumulate in the queue and wait for processing, eventually leading to timeout failure. In this case, it is meaningless to send the request to the server, because the caller has determined that the request has failed and no longer cares about the response data.

Figure 4-4d Request timeout

Therefore, ExecutorAsync should remove invalid requests from the queue in time, while the remaining requests are still "fresh". This strategy can forcibly shorten the queue length to reduce the accumulation time of subsequent requests in the queue and prevent request timeouts. At the same time, this strategy can also save memory and IO overhead by avoiding storage and sending invalid requests.

Figure 4-4e ExecutorAsync feature: discard

batch request

In the previous section, we saw the scheduling strategy of ExecutorAsync, including queuing, batching, concurrency, and discarding. As shown in the code below, ExecutorAsync only needs to provide the submit(Request) method to the outside for submitting a single request. The request is represented by the data object Request, and its field Request.resultPromise is a Promise object, used to notify the response data; in scenarios where exception handling is required, we use PromiseOrException<T, Exception> as the realization of Promise, and the generic template T is modified Is the actual type of response data.

public class ExecutorAsync {

    public PromiseOrException<T, Exception> submit(Request<T> request) {
        return request.resultPromise;
    }
}

Next, let's take a look at the implementation principle of ExecutorAsync. Because the source code is more detailed and longer, this section uses a flowchart to explain the higher-level design, as shown in Figure 4-5.

Figure 4-5 Principle of ExecutorAsync

1. Submit request . The caller calls ExecutorAsync.submit(Request), and each call submits a request. The request is stored in the queue, waiting for subsequent scheduling execution. The structure of the parameter Request is shown in the following code, including the following fields:

predicate: function to determine whether the request is valid, invalid requests (such as timed out requests) will be discarded. See step 2 for details.

resultPromise: Notification response data.

public class Request<Response> {
    public final PredicateE predicate; 
    public final PromiseOrException<Response, Exception> resultPromise;
}

2. At regular intervals, or when queue.size() reaches the bulk size, try to assemble the batch request . Requests are taken out from the queue in turn, and the function Request.predicate is executed for each request to determine whether the request is still to be submitted; the number of valid requests taken out does not exceed the bulk size.
Predicate is a function, similar to the jdk Predicate interface, in the form shown in the following code. The interface function test() can return normally, indicating that the request is still valid; it can also throw an exception, indicating the reason for the invalid request, such as waiting timeout. If an exception is thrown, the request is directly discarded, and the occurrence of the exception will be notified to Request.resultPromise, so that the caller can execute the exception handling logic.

public interface PredicateE {
    void test() throws Exception;
}

3. Submit a batch request . The second step is to take out at most bulk size requests from the queue and call RequestFunc.execute(requests) with them as parameters to submit batch requests. The form of the interface RequestFunc is shown in the following code. The interface method execute(requests) takes several requests as parameters, assembles them into batch requests, and calls the underlying database API to submit them.

public interface RequestFunc<T>{
    void execute(List<Request<T>> requests);
}

4. After receiving the response, for each request, notify Request.resultPromise of the response data .

5. In order to prevent the server from overloading, ExecutorAsync can limit the number of concurrent requests not to exceed parallelism . We set the count variable inFlight=0 to count the number of batch requests being executed:

a. When trying to assemble a batch request (step 2), first judge inFlight<parallelism, and only when the conditions are met can the requests to be executed be taken out of the queue.
b. After submitting a batch request (step 3, RequestFunc.execute()), inFlight++.
c. When a batch of requests receives response data (step 4, Request.resultPromise is notified), inFlight--. At this time, if the number of requests in the queue still exceeds the bulk size, go back to step 2, and then take out a batch of requests to execute.
In summary, ExecutorAsync uses a queue queue to temporarily store pending requests; when a batch request needs to be submitted, it uses PredicateE to filter valid requests and discards invalid requests; for a batch of requests, call RequestFunc.execute() to submit batches and receive responses Request.resultPromise notification after data. The above process satisfies the constraints to prevent the server from overloading: the number of batch requests is at most bulk size; the number of batch requests being executed at the same time does not exceed parallelism. The above is the basic principle of ExecutorAsync; the actual application also needs to deal with details such as configuration parameters, generic templates, etc. This article will not explain it in detail because of space limitations.

5. Summary

This article introduces the asynchronous model and Promise design pattern , discusses the design principles of asynchronous API, and introduces corresponding solutions. Asynchronous mode is not only a simple process of "submitting a request-processing a response", sometimes it also requires designing an exception handling mechanism and scheduling based on the relationship between requests. When dealing with these complex scenarios, the API needs to maintain a purely asynchronous feature, and cannot block during the process of submitting requests and processing responses; it is necessary to make full use of compile-time checks to prevent callers from missing branches, especially inevitable abnormal branches; API needs to encapsulate complex and repetitive implementation details, and try to keep the caller's code structure concise and easy to understand.

This series of articles aims to popularize the asynchronous mode, hoping to play a role in inducing jade, help readers understand the basic principles of the asynchronous mode, have an understanding of the actual problems that may be encountered, and initially explore the implementation mechanism of the asynchronous mode. However, the actual projects and tools are far more complex than those introduced in this article. Readers are also invited to do research and selection work. Looking at the various existing asynchronous APIs, readers will find that there is still no unified standard for asynchronous mode. Taking the database client as an example, the functional forms of various asynchronous APIs are not the same. Both listener API and Promise API are used. Some APIs are asynchronous in form, but in some cases blocking will occur; the bottom layer of asynchronous API Some are based on thread pool/connection pool, and some are based on responsive models (such as netty). Therefore, readers must fully understand the API form, blocking characteristics, and threading model of asynchronous tools before they can be used in the project; if the existing tools do not meet the development specifications, they can boldly encapsulate or implement the required features by themselves. Regarding the encapsulation and implementation of asynchronous tools, readers are also welcome to communicate and correct me.