Let's talk about graceful closure from beginning to end

gracefully close (Graceful Shutdown/Graceful Exit), this term does not seem to have any official definition, and no authoritative source has been found. However, when searching for Graceful Exit in Bing, the second article that appears is specifically for women to deal with divorce. Website...
Good guy, a one-stop solution for women's divorce, which is too professional. It seems that not only the program needs to be closed gracefully, but even the divorce needs to be Graceful!

In a computer, graceful shutdown actually refers to a program shutdown scheme. Since there is a graceful closure, there must be an inelegant closure.

Graceful shutdown of Windows

Take the Windows computer power on and off, long press the power button to force the shutdown, or directly power off the shutdown, this is a hard shutdown (hard shutdown), the operating system does not receive any signal and disappears directly, it is not elegant !

At this time, the system or some software has not been processed before closing. For example, you have worked overtime to write a PPT for 4 hours and have not had time to save...

But in general, except for crashes, few people force shut down. Most people still use the power option -> shutdown operation to let the operating system handle the shutdown itself. For example, Windows will actively close all applications before shutting down, but many applications will capture the shutdown event of the process, causing it to fail to shut down normally, thus causing the system to fail to shut down normally. For example, in the office suite, if you don't save before closing, a pop-up will pop up for you to save, and this mechanism will interfere with the normal shutdown of the operating system.

Or if you are using Win10, you can update the system by yourself at any time. If you power off when you update the system and force a shutdown, you may have a surprise when you turn it on again... The update file is half written, you guessed it. What's the problem?

Graceful closure in the network

The network is unreliable!

I believe everyone has memorized TCP’s eight-legged essay, and the connection can only be disconnected after four waves of hands, but the four waves of hands are also based on the premise of normal shutdown. If you forcibly unplug the network cable or force the power off, the opposite end cannot detect your disconnection in time. At this time, if the opposite end continues to send messages, it will receive an error.

You see, in addition to the four graceful waves, there is also TCP KeepAlive doing the heartbeat. This is not enough. The application layer has to do another heartbeat. At the same time, it must correctly and gracefully handle the connection disconnection, Connection Reset and other errors.

Therefore, if we are writing a network program, we must provide a shutdown mechanism to shut down the socket/server normally in the shutdown event, thereby reducing more abnormal problems caused by the shutdown.

How to monitor the shutdown event?

Various languages will provide a monitoring mechanism for this shutdown event, but the usage is different. With the help of this shutdown monitor, it is easy to achieve graceful shutdown.

JAVA monitoring is off

JAVA provides a simple shutdown event monitoring mechanism, which can receive normally closes the signal, such as the Ctrl+C exit signal under the command line program.

Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
    @Override
    public void run() {
        System.out.println("Before shutdown...");
    }
}));

After this configuration is completed and before the normal shutdown, the ShutdownHook thread will be started and executed, and the output Before shutdown will be output. Of course, if you force the shutdown directly, such as the end process under Windows, Kill -9 under Linux...the gods will not be able to monitor

Close monitoring in C++

There is a similar implementation in C++, as long as the function is registered in the atexit function, the registered fnExit function can be executed before the program is normally closed.

void fnExit1 (void)
{
  puts ("Exit function 1.");
}

void fnExit2 (void)
{
  puts ("Exit function 2.");
}

int main ()
{
  atexit (fnExit1);
  atexit (fnExit2);
  puts ("Main function.");
  return 0;
}

Problems that may be encountered during shutdown

Imagine such a scenario, a message consumption logic, after the transaction is successfully submitted, it is pushed to the peripheral system. The shutdown signal has been received a long time ago, but due to the accumulation of a large number of messages, some of them have been accumulated in the memory queue, but the logic of parallel consumption processing has not been executed.
​

At this time, some consumer threads submit transactions, and they receive the Force Kill signal before pushing the peripheral system. Then there will be data inconsistencies. The service data has been in the database, but the three parties have not been pushed...

To give another example of a database, the storage engine has the concepts of clustered index and non-clustered index. If a clustered index is written just after an Insert statement is executed, the process is killed before the non-clustered index can be written, then these two indexes The data is directly inconsistent!
​

However, as a storage engine, it will definitely deal with this inconsistency. But if it can be shut down normally and the storage engine is executed safely, the risk of this inconsistency will be greatly reduced.

Process stopped

The mechanism for stopping the JAVA process is that after all non-daemon threads have stopped , the process will exit. Then can the process be closed by sending a shutdown signal directly to the JAVA process? Certainly not!
​

The threads in JAVA are non-blocking threads by default. Non-daemon threads will not stop as long as the JVM process will not stop. So after receiving the shutdown signal, you have to shut down all threads by yourself, such as the thread pool...

Thread interruption

How to actively close the thread? Sorry, this really can't be closed (the stop method has been obsolete since JAVA 1.1), you can only wait for the thread to finish executing by itself, or implement it through soft state plus interrupt:

private volatile boolean stopped = false;

@Override
public void run() {
    while (!stopped && Thread.interrupted()){
        // do sth...
    }
}

public void stop(){
    stopped = true;
    interrupt();
}

When the thread is in the WAITTING state, the interrupt method will interrupt the WAITTING state, force a return and throw InterruptedException . For example, when our thread is stuck in the Socket Read operation, or some lock waiting state under Object.wait/JUC, calling the interrupt method will interrupt the waiting state and throw an exception directly.
​

But if the thread is not stuck in the WAITING state, and it is still created in the thread pool and has no soft state, then the above shutdown strategy may not be applicable.

Thread pool shutdown strategy

ThreadPoolExecutor provides two shutdown methods:

1. shutdown -interrupt Idle Worker thread, waiting for the completion of all tasks (threads). Because idle worker threads will be in the WAITING state, the interrupt method will directly interrupt the WAITING state and stop these idle threads.
2. shutdownNow -interrupt All Worker threads, whether idle or not. For idle threads, just like the shutdown method, they are directly stopped. For the worker threads that are working, they are not necessarily in the WAITING state, so interrupt cannot be guaranteed to be closed.

Note: Most thread pools, or frameworks that call thread pools, their default shutdown strategy is to call shutdown instead of shutdownNow, so the executing thread will not necessarily be Interrupt
​

But as a business thread, it must process **InterruptedException** . Otherwise, in case there is shutdownAll, or the interruption of manually created threads, the business thread does not respond in time, which may cause the thread to be completely unable to shut down

The closing strategy of the tripartite framework

In addition to the thread pool of the JDK, some third-party frameworks/libraries also provide some methods for graceful shutdown.

• EventLoopGroup.shutdownGracefully/shutdown in Netty-Shut down the thread pool and other resources
• Redisson.shutdown in Reddsion-close the connection of the connection pool, destroy various resources
• CloseableHttpClient.close in Apache HTTPClient-close the connection of the connection pool, close the Evictor thread, etc.

These mainstream mature frameworks will provide you with a graceful shutdown method to ensure that after you call shutdown, it can destroy resources, and shuts down the thread/pool created by it.
​

In particular, this kind of tripartite framework that involves creating threads must provide a way to shut down normally, otherwise it may happen that the thread cannot be shut down, causing the JVM process to fail to exit normally.

Graceful shutdown in Tomcat

Tomcat's shutdown script (sh version) is well designed, and I will tell you how to shut it down directly:

commands:
    stop              Stop Catalina, waiting up to 5 seconds for the process to end
    stop n            Stop Catalina, waiting up to n seconds for the process to end
    stop -force       Stop Catalina, wait up to 5 seconds and then use kill -KILL if still running
    stop n -force     Stop Catalina, wait up to n seconds and then use kill -KILL if still running

This design is very flexible and directly provides 4 closing methods for you to choose at will.

In force mode, a SIGTERM Signal (kill -15) will be sent to the process. This signal can be captured by the JVM, and the registered ShutdownHook thread will be executed. After waiting for 5 seconds, if the process is still there, Force Kill, the process is shown in the following figure:

​

ShutdownHook thread registered in Tomcat will be executed, manually closing various resources, such as Tomcat's own connection, thread pool and so on.
​

Of course, there is the most important step, closing all apps:

// org.apache.catalina.core.StandardContext#stopInternal

// 关闭所有应用下的所有 Filter - filter.destroy();
filterStop();
// 关闭所有应用下的所有 Listener - listener.contextDestroyed(event);
listenerStop();

With the help of these two hooks before closing, the application can handle the closing by itself, such as the Servlet Context Listener used in the XML era:

<listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
</listener>

In this Listener, Spring calls the closing method of Application Context by itself:

public void contextDestroyed(ServletContextEvent event) {
    // 关闭 Spring Application Context
    this.closeWebApplicationContext(event.getServletContext());
    ContextCleanupListener.cleanupAttributes(event.getServletContext());
}

Spring's graceful shutdown

After Spring ApplicationContext executes close, Spring will destroy all Beans. As long as your Bean is configured with the destroy strategy or implements the AutoCloseable interface, then Spring can call destroy when destroying the Bean, such as the thread pool wrapped by Spring. ThreadPoolTaskExecutor , it implements the DisposableBean interface:

// ThreadPoolTaskExecutor
public void destroy() {
    shutdown();
}

shutdown Bean, the thread pool will execute 0619af2af0e429, and you do not need to manually control the shutdown of the thread pool.
​

It should be noted here that Spring Bean creation and Bean destruction is reversed:
​

Use the reverse order when destroying, you can ensure that the dependent Bean can be destroyed normally, and will not be destroyed in advance. For example, in the dependency relationship A->B->C, we must ensure that C is loaded first; then, if C is destroyed first, B may still be running, and B may report an error at this time.
​

Therefore, when dealing with beans with complex dependencies, the pre-Bean should be loaded first, and the basic Beans such as the thread pool should be loaded last. When it is destroyed, the basic Bean of the thread pool will be destroyed first.

---

Most frameworks/libraries that need to be closed normally will integrate the destruction entry of Spring Bean when integrating Spring.
​

For example, Redis client-Lettuce, spring-data-redis provides the integration of lettuce. The integration class LettuceConnectionFactory directly implements the DisposableBean interface and is closed inside the destroy method.

// LettuceConnectionFactory 

public void destroy() {
    this.resetConnection();
    this.dispose(this.connectionProvider);
    this.dispose(this.reactiveConnectionProvider);

    try {
        Duration quietPeriod = this.clientConfiguration.getShutdownQuietPeriod();
        Duration timeout = this.clientConfiguration.getShutdownTimeout();
        this.client.shutdown(quietPeriod.toMillis(), timeout.toMillis(), TimeUnit.MILLISECONDS);
    } catch (Exception var4) {
        if (this.log.isWarnEnabled()) {
            this.log.warn((this.client != null ? ClassUtils.getShortName(this.client.getClass()) : "LettuceClient") + " did not shut down gracefully.", var4);
        }
    }

    if (this.clusterCommandExecutor != null) {
        try {
            this.clusterCommandExecutor.destroy();
        } catch (Exception var3) {
            this.log.warn("Cannot properly close cluster command executor", var3);
        }
    }

    this.destroyed = true;
}

The same is true for other frameworks. When integrating Spring, resources are destroyed based on Spring's destroy mechanism.

The problem of Spring destruction mechanism

Now there is such a scenario, we have created a certain MQ consumer client object, let's call it XMQConsumer. In this consumer client, a thread pool is built in, and when a message is pulled, it will be thrown into the thread pool for execution.
​

In the code consumed by the message MQ, the database connection pool-DataSource is needed, and the HTTP request-HttpClient needs to be sent, both of which are managed by Spring. However, the loading order of the two Beans, DataSource and HttpClient, is relatively high. When the XMQConsumer is started, these two Beans can be used after initialization.
​

However, the destroy-method is not specified for this XMQConsumer, so when the Spring container is closed, the consumer client will not be closed, and the consumer client will continue to pull messages and consume messages.
​

At this time, when Tomcat receives the shutdown signal, according to the shutdown process above, Spring will destroy the reverse order

Since XMQConsumer did not specify destroy , Spring will only destroy #2 and #3 two beans. However, the threads in the XMQConsumer thread pool and the main thread are asynchronous. When the first two objects are destroyed, the consumer thread is still running. During the operation, the database needs to be operated and the request needs to be sent through HttpClient. At this time, it will appear: XXX is Closed Class of errors.

Spring Boot gracefully close

After Spring Boot, this shutdown mechanism has changed a little. Because the Spring project was deployed and run in Tomcat before, Tomcat will start Spring.
​

But in Spring Boot (Executeable Jar method), the order is reversed, because Spring is started directly, and then Tomcat (Embedded) is started in Spring. The startup method has changed, and the shutdown method must have also changed. shutdownHook , and finally Spring shuts down Tomcat.
​

As shown in the figure below, this is the start/stop sequence of the two methods:

K8S closes gracefully

What I'm talking about here is the mechanism of K8S gracefully closing the POD, similar to the Tomcat shutdown script introduced above, both send the SIGTERM Signal first, and if the process is still in N seconds, then Force Kill.
​

It's just that the initiator of Kill has become K8S/Runtime. When the container is running, it will send Kill(TERM) signals to the main processes of all containers in the Pod:

Similarly, if within the grace period ( terminationGracePeriodSeconds , default 30 seconds), the process in the container does not complete the shutdown logic, the process will be forcibly killed.

When K8S encounters SpringBoot (Executeable Jar)

Nothing special, K8S sends a TERM signal to the Spring Boot process, and then executes Spring Boot's ShutdownHook

When K8S meets Tomcat

It's catalina.sh , except that the initiator of the shutdown becomes K8S

Summarize

After talking about so many graceful closures, how can it be considered graceful? Here is a brief summary of 3 points:

1. As a framework/library, it is necessary to provide a method for normal shutdown, manually closing the thread/thread pool, destroying connection resources, FD resources, etc.
2. As an application, you must handle InterruptedException well, don't ignore this exception, otherwise there is a risk that the process will not exit normally
3. When closing, we must pay attention to the order, especially the resources of the thread pool, and we must ensure that the thread pool is closed first. The safest way is not to interrupt the thread, wait for the thread to complete its execution, and then shut down.

refer to

• https://kubernetes.io/zh/docs/concepts/workloads/pods/pod-lifecycle/
• https://github.com/apache/tomcat
• https://whatis.techtarget.com/definition/graceful-shutdown-and-hard-shutdown
• https://www.wikiwand.com/en/Graceful_exit
• https://docs.spring.io/spring-boot/docs/2.3.0.RELEASE/reference/html/spring-boot-features.html#boot-features-graceful-shutdown