Build an efficient and scalable result cache

Hello, I am crooked.

I'm here to fill the hole.

Released last week "When Synchronized encounters this thing, there is a big pit, pay attention! this article.

At the end of the article, I mentioned the content of Section 5.6 of "Java Concurrent Programming in Practice" and said that everyone can go and see it.

I don't know how many students have watched it, but I know that most of the students have not watched it, so I will also arrange for you in this article, how to better implement a caching function.

Feel the evolution of the master's code scheme.

need

Isn't it mid-February, and the postgraduate entrance examination results will be released soon, so let me take this as an example.

The requirements are very simple: query from the cache, get it from the database if it cannot be found, and put it in the cache for the next use.

The core code is probably like this:

Integer score = map.get("why");
if(score == null){
   score = loadFormDB("why");
   map.put("why",score);
}

With the core code, after I complete the code, it should look like this:

public class ScoreQueryService {

    private final Map<String, Integer> SCORE_CACHE = new HashMap<>();

    public Integer query(String userName) throws InterruptedException {
        Integer result = SCORE_CACHE.get(userName);
        if (result == null) {
            result = loadFormDB(userName);
            SCORE_CACHE.put(userName, result);
        }
        return result;
    }

    private Integer loadFormDB(String userName) throws InterruptedException {
        System.out.println("开始查询userName=" + userName + "的分数");
        //模拟耗时
        TimeUnit.SECONDS.sleep(1);
        return ThreadLocalRandom.current().nextInt(380, 420);
    }
}

Then make a main method to test it:

public class MainTest {

    public static void main(String[] args) throws InterruptedException {
        ScoreQueryService scoreQueryService = new ScoreQueryService();
        Integer whyScore = scoreQueryService.query("why");
        System.out.println("whyScore = " + whyScore);
        whyScore = scoreQueryService.query("why");
        System.out.println("whyScore = " + whyScore);
    }
}

Run the code:

Good guy, I got a score of 408 in the first round. If I can really get this score in the postgraduate entrance examination, I am afraid that I will wake up laughing from a dream.

The demo is very simple, but please note that I'm about to start deforming.

First modify the main method to look like this:

public class MainTest {

    public static void main(String[] args) {
        ExecutorService executorService = Executors.newFixedThreadPool(5);
        ScoreQueryService scoreQueryService = new ScoreQueryService();
        for (int i = 0; i < 3; i++) {
            executorService.execute(()->{
                try {
                    Integer why = scoreQueryService.query("why");
                    System.out.println("why = " + why);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            });
        }
    }
}

Use the thread pool to submit tasks and simulate three query requests at the same time. Since the loadFormDB method has simulated time-consuming operations, these three requests will not get data from the cache.

Is that exactly the case?

Take a look at the running results:

The output is three times, and three different scores are obtained, indicating that the loadFormDB method is indeed executed three times.

Okay, classmates, so here comes the question.

Obviously, in this scenario, I only want one thread to execute the loadFormDB method, so what should I do?

The moment you see this question, I wonder if there is a flash in your head that reminds you of the cache problem three combos: cache avalanche, cache breakdown, and cache penetration.

After all, one of the solutions to cache breakdown is to only need one request thread to build the cache, and other threads to poll and wait.

Then the solution of Redis distributed lock naturally appeared in my mind, and even thought that the setNX command should be used to ensure that only one thread is released successfully. An imperceptible smile leaked from the corner of his mouth, and he even wanted to close this article.

Sorry, put away your smile, you can't use Redis, you can't use third-party components, you can only use JDK stuff.

Don't ask why, just ask that it wasn't introduced.

What do you do at this time?

initial plan

After hearing that you can't use third-party components, you didn't panic at all, and shouted: key here.

With the keyboard, you only need to click three times to write the code:

Adding a synchronized keyword is done, and you even remember the programmer's self-cultivation, completed a wave of self-tests, and found that there is no problem:

The loadFromDB method is executed only once.

However, friends, have you ever thought that the granularity of your lock is a bit too large.

Just lock the whole method.

Originally a good parallel method, you click it and make it serial:

And you are killing indiscriminately. For example, in the above diagram, if you say that when you query the results of why for the second time, it is understandable to block this request.

But you also intercepted the results of the first query mx. It made the mx classmates bewildered and couldn't figure out what was going on.

Note that at this time, it is natural to think of reducing the granularity of locks, changing the scope of locks from global to local, and come up with solutions such as using why objects as locks.

For example, the pseudo code is changed to this:

Integer score = map.get("why");
if(score == null){
   synchronized("why"){
       score = loadFormDB("why");
       map.put("why",score);
   }    
}

If you still haven't reacted here, then I'll give another example.

Suppose my query condition here changes to the number of Integer type?

For example, my number is 200. Is the pseudo code like this:

Integer score = map.get(200);
if(score == null){
   synchronized(200){
       score = loadFormDB(200);
       map.put(200,score);
   }    
}

If you haven't reacted to this, I can only shout: You fake reader! Surely you didn't read the previous article, did you?

The previous "When Synchronized encounters this thing, there is a big pit, pay attention! 》 n't this whole article about this?

If you don't know what the problem is, go look it up.

This article is definitely not going in this direction. You can't keep staring at synchronize, or you won't be able to open your mind.

We can't use the synchronized thing here.

But if you take a closer look, if you don't need to synchronize, this map will not work:

private final Map<String, Integer> SCORE_CACHE = new HashMap<>();

This is a HashMap, not thread-safe.

How to do?

Evolve.

evolve

This step is very simple, compared with the original program, just replace HashMap with ConcurrentHashMap.

Then nothing was done.

Do you feel a little confused, or even feel that you must have been tricked?

That's right, because this step of change is a plan in the book. When I first saw it, I felt a little confused anyway:

I really didn't lie to you, if you don't believe me, I will take a picture for you to see:

.png)

Compared with the original scheme, the only advantage of this scheme is that the degree of concurrency increases, because ConcurrentHashMap is thread-safe.

However, as a cache, it can be seen from the above schematic diagram that it is just one sentence: eggs are useless.

Because it simply cannot meet the requirement of "under the same request, if there is no cache, only one request thread executes the loadFormDB method", such as why the loadFormDB method is executed twice in two query operations in a short period of time.

What's wrong with it?

If multiple threads are checking the results of this person, if one thread executes the loadFormDB method, and the other threads do not perceive that there are threads executing the method at all, then they rush in and take a look: I will go, Nothing in the cache at all? Then I will also execute the loadFormDB method.

The calf is finished, and the execution is repeated.

So is there a mechanism in the JDK native method to indicate that there is already a thread requesting the why score query is executing the loadFormDB method, then other threads querying the why score can just wait for the result, and there is no need to execute it yourself again.

This is the time to test your knowledge.

What did you think of?

continue to evolve

FutureTask is a very important part of asynchronous programming.

For example, in the application of thread pool, when you submit a task using the submit method, its return type is Future:

Anyway, based on the Future, you can play with flowers.

For example, in our scenario, if FutureTask is used, then our Map needs to be modified as follows:

Map<String, Future<Integer>> SCORE_CACHE = new ConcurrentHashMap<>();

We do this by maintaining the relationship between name and Future.

Future itself represents a task. For the requirement of cache maintenance, it does not care whether the task is in execution or has been executed. This "it" refers to the Map of SCORE_CACHE.

For Map, as long as there is a task into it.

Whether the task is executed or not should be concerned by the thread from the get in the Map to the corresponding Future.

How does it care?

By calling the Future.get() method.

The whole code is written like this:

public class ScoreQueryService {

    private final Map<String, Future<Integer>> SCORE_CACHE = new ConcurrentHashMap<>();

    public Integer query(String userName) throws Exception {
        Future<Integer> future = SCORE_CACHE.get(userName);
        if (future == null) {
            Callable<Integer> callable = () -> loadFormDB(userName);
            FutureTask futureTask = new FutureTask<>(callable);
            future = futureTask;
            SCORE_CACHE.put(userName, futureTask);
            futureTask.run();
        }
        return future.get();
    }

    private Integer loadFormDB(String userName) throws InterruptedException {
        System.out.println("开始查询userName=" + userName + "的分数");
        //模拟耗时
        TimeUnit.SECONDS.sleep(1);
        return ThreadLocalRandom.current().nextInt(380, 420);
    }
}

I'm afraid you are not familiar with futureTask, so I briefly explain the four lines of code about futureTask, but I still strongly recommend that you master this thing, after all, it is not an exaggeration to say that it is one of the cornerstones of asynchronous programming.

The cornerstone still has to be grasped and understood, otherwise it is easy to be grasped by the interviewer.

Callable<Integer> callable = () -> loadFormDB(userName);
FutureTask futureTask = new FutureTask<>(callable);
futureTask.run();
return future.get();

First I built a Callable as an input to the FutureTask constructor.

The above description of the constructor translates as: Create a FutureTask that will execute the given Callable at runtime.

The "runtime" refers to the line of code futureTask.run() , and the "given Callable" is the loadFormDB task.

That is to say, after calling futureTask.run() , it is possible to execute the loadFormDB method.

Then calling future.get() is to get the result of the Callable, that is, get the result of the loadFormDB method. If the method has not finished running, just die and so on.

Here's what the book says about this plan:

Mainly focus on the part I underlined, I say sentence by sentence

It has only one flaw, which is that there is still a loophole where two threads compute the same value.

This sentence is actually easy to understand, because there is always an action of "①acquisition-②judgment-③put" in the code.

This action is not atomic, so there is a certain chance that both threads will rush in, and then find that there is nothing in the cache, so they both go to the if branch.

But the places marked ① and ② are definitely indispensable from the perspective of requirement realization.

The only place where we can think of a way is the place marked with ③.

What can be done?

Don't worry, in the next section, I will explain the second half of the sentence first:

The probability of occurrence of this vulnerability is much smaller than the probability of occurrence in Memoizer2.

Memoizer2 refers to the previous scheme after replacing HashMap with ConcurrentHashMap.

So why is the probability of triggering the bug just mentioned smaller than the previous solution after the introduction of Future?

The answer lies in these two lines of code:

Before, it was necessary to complete the execution of the business logic, and only after the return value could be maintained in the cache.

Now, the cache is maintained first, and then the business logic is executed, which saves the time of executing the business logic.

Generally speaking, the most time-consuming place is the execution of business logic, so this "much less than" comes from this.

then what should we do?

Then evolve.

final version

In the book, when dealing with the non-atomic action of "add if there is none" above, a method of map is mentioned:

.png)

Map's putIfAbsent, this method is amazing. Take a look at:

First of all, we can know from the place labeled ① that if the key passed in by this method has not been associated with a value (or is mapped to null), it will be mapped with the given value and return null, otherwise it will return the current value.

If we only care about the return value, that is: if there is, return the corresponding value, if not, return null.

What does the place marked ② say?

It says that the default implementation makes no guarantees about the synchronization or atomicity of this method. If you want to provide atomicity guarantee, then please override this method and write it yourself.

So, let's take a look at how this method of ConcurrentHashMap works:

The atomicity is guaranteed by the synchronized method. When the same key is operated, only one thread is guaranteed to perform the put operation.

So the final implementation given in the book is as follows:

public class ScoreQueryService {

    public static final Map<String, Future<Integer>> SCORE_CACHE = new ConcurrentHashMap<>();

    public Integer query(String userName) throws Exception {
        while (true) {
            Future<Integer> future = SCORE_CACHE.get(userName);
            if (future == null) {
                Callable<Integer> callable = () -> loadFormDB(userName);
                FutureTask futureTask = new FutureTask<>(callable);
                future = SCORE_CACHE.putIfAbsent(userName, futureTask);
                //如果为空说明之前这个 key 在 map 里面不存在
                if (future == null) {
                    future = futureTask;
                    futureTask.run();
                }
            }
            try {
                return future.get();
            } catch (CancellationException e) {
                System.out.println("查询userName=" + userName + "的任务被移除");
                SCORE_CACHE.remove(userName, future);
            } catch (Exception e) {
                throw e;
            }
        }
    }

    private Integer loadFormDB(String userName) throws InterruptedException {
        System.out.println("开始查询userName=" + userName + "的分数");
        //模拟耗时
        TimeUnit.SECONDS.sleep(5);
        return ThreadLocalRandom.current().nextInt(380, 420);
    }
}

There are three differences from the previous scheme.

• The first is to replace the put method with putIfAbsent.
• The second is the addition of a while(true) loop.
• The third is that future.get() performs the action of clearing the cache after throwing a CancellationException.

The first one has nothing to say, it has been explained before.

The second and third, to be honest, when they were used together, I didn't really understand it.

First of all, programmatically, these two are complementary codes, since the while(true) loop I understand only works when future.get() throws a CancellationException.

Throws a CancellationException exception, indicating that the current task has been called by the cancel method elsewhere, and due to the existence of while(true) and the current task has been removed, so the if condition is successful, it will build the same task again, Then continue to execute:

That is to say, the tasks removed and the tasks put in are exactly the same.

Shouldn't it be removed?

It doesn't matter if you haven't turned the corner, I'll show you the code first, and you'll understand:

Among them, the code of ScoreQueryService has been given before, so I will not take a screenshot.

It can be seen that this time only one task was thrown into the thread pool, and then the task in the cache was taken out, and the cancel method was called to cancel it.

The output of this program is this:

Therefore, due to the existence of while(true), the cancel method fails.

Then I said earlier: the tasks removed are the same as the tasks put in. Shouldn't it be removed?

It's like this in the code:

I don't know why the author has a special removal action. After this wave of analysis, this line of code can be commented out.

but...

Is it right?

That's not right, old man. If this line of code is commented out, the output of the program would look like this:

It's become an endless loop.

Why has it become an infinite loop?

Because FutureTask has a life cycle:

After being canceled, the life cycle is completed, so if it is not removed from the cache, then Barbie Q will be taken out. The one that is taken out is always the one that was canceled, then an exception will be thrown and the cycle will continue.

This is how the cycle of death comes about.

So the action to remove must be there, while(true) depends on your needs, plus it means that the cannel method "fails", and if you remove it, you can call the cannel method.

About FutureTask If you are not familiar with it, I have written two articles, you can take a look.

"Father, look at this code, kneel!

"The BUG written by Doug Lea in the JUC bag was discovered by netizens again.

Next, let's verify that the final code is working correctly:

The scores finally found by the three threads are the same, and there is nothing wrong.

If you want to observe the blocking situation, you can lengthen the sleep time a bit:

Then, run the code and see the stack information:

One thread is sleeping, and the other two threads execute the get method of FutureTask.

Good understanding of sleep, why are the other two threads blocked on the get method?

Very simple, because the future returned by the other two threads is not null, which is determined by the characteristics of the putIfAbsent method:

Well, the code of the final solution given in the book is also explained.

But there are still two "pits" left in the book:

.png)

One is that the cache expiration mechanism is not supported.

One is that the cache elimination mechanism is not supported.

Wait a minute, let's talk about my other plan first.

There is another plan

In fact, I also have a plan, and I will show it to you:

public class ScoreQueryService2 {

    public static final Map<String, Future<Integer>> SCORE_CACHE = new ConcurrentHashMap<>();

    public Integer query(String userName) throws Exception {
        while (true) {
            Future<Integer> future = SCORE_CACHE.get(userName);
            if (future == null) {
                Callable<Integer> callable = () -> loadFormDB(userName);
                FutureTask futureTask = new FutureTask<>(callable);
                FutureTask<Integer> integerFuture = (FutureTask) SCORE_CACHE.computeIfAbsent(userName, key -> futureTask);
                future = integerFuture;
                integerFuture.run();
            }
            try {
                return future.get();
            } catch (CancellationException e) {
                SCORE_CACHE.remove(userName, future);
            } catch (Exception e) {
                throw e;
            }
        }
    }

    private Integer loadFormDB(String userName) throws InterruptedException {
        System.out.println("开始查询userName=" + userName + "的分数");
        //模拟耗时
        TimeUnit.SECONDS.sleep(1);
        return ThreadLocalRandom.current().nextInt(380, 420);
    }
}

The difference from the solution given in the book is that putIfAbsent is replaced by computeIfAbsent:

V computeIfAbsent(K key, Function<? super K, ? extends V> mappingFunction)

computeIfAbsent, first of all, it is also a thread-safe method. This method will check the Key in the Map. If it finds that the Key does not exist or the corresponding value is null, it will call the Function to generate a value, then put it into the Map, and finally return this value; otherwise, returns the value that already exists in the Map.

putIfAbsent, if the Key does not exist or the corresponding value is null, set the Value in, and then return null; otherwise, only return the corresponding value in the Map without doing other operations.

So one of the differences between the two is in the return value.

After using computeIfAbsent, the same FutureTask is returned every time, but due to the life cycle of FutureTask, or the existence of state reversal, even if all three threads call its run method, this FutureTask will only be executed successfully once .

You can take a look at the source code of this run method. As soon as it comes in, it is the judgment of the status and the current operation thread:

So after executing the run method once, calling the run method again will not actually execute.

But from an elegant point of view of program implementation, the putIfAbsent method is better.

What about the pit?

Didn't I say that the final solution has two pits:

• One is that the cache expiration mechanism is not supported.
• One is that the cache elimination mechanism is not supported.

Under the premise of using ConcurrentHashMap, if these two features are to be supported, corresponding development needs to be carried out, such as the introduction of timed tasks to solve them, which is troublesome to think about.

At the same time, I also thought of spring-cache. I know that there is a ConcurrentHashMap as a cache implementation.

I want to see how this component solves these two problems.

Without further ado, let me pull down the code and take a look:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

Since spring-cache is not the focus of this article, I will directly talk about the source code of the key places.

As for how to find it here, I will not introduce it in detail. I will arrange an article to explain it in detail later.

In addition, I have to say: spring-cache is really an elegant comparison, whether it is the application of source code or design pattern, it is very good.

First, we can see that there is a parameter called sycn in the @Cacheable annotation, and the default value is false:

Regarding this parameter, the explanation on the official website is as follows:

https://docs.spring.io/spring-framework/docs/current/reference/html/integration.html#cache-annotations-cacheable-cache-resolver

It is a solution to the situation of how to maintain the cache we mentioned earlier. The method of use is also very simple.

The source code of the core part corresponding to this function is in this location:

org.springframework.cache.interceptor.CacheAspectSupport#execute(org.springframework.cache.interceptor.CacheOperationInvoker, java.lang.reflect.Method, org.springframework.cache.interceptor.CacheAspectSupport.CacheOperationContexts)

In the above method, it will be judged whether it is a sync=true method, and if so, it will enter the if branch.

Then it will execute the following important method:

org.springframework.cache.interceptor.CacheAspectSupport#handleSynchronizedGet

In this method, the parameter cache is an abstract class, and Spring provides six default implementations:

And what I care about is the implementation of ConcurrentMapCache, click in and have a look, good guy, I am familiar with this method:

org.springframework.cache.concurrent.ConcurrentMapCache#get

The computeIfAbsent method, we just talked about it. But when I turned left and right, I couldn't find the place to set the expiration time and elimination strategy.

So, I went to the official website again, and found that the answer was written directly on the official website:

https://docs.spring.io/spring-framework/docs/current/reference/html/integration.html#cache-specific-config

As mentioned here, what the official provides is an abstraction of caching, not a concrete implementation. Cache expiration and retirement mechanisms are outside the scope of the abstraction.

why?

For example, take ConcurrentHashMap as an example. Suppose I provide an abstraction of cache expiration and elimination mechanism. How do you think ConcurrentHashMap implements this abstract method?

It cannot be implemented because it does not support this mechanism in the first place.

Therefore, officials believe that such functions should be implemented by specific cache implementation classes rather than providing abstract methods.

This also answers the two questions raised by the previous final plan:

• One is that the cache expiration mechanism is not supported.
• One is that the cache elimination mechanism is not supported.

Don't ask, asking is that the native method cannot support it. If you want to implement your own write code, or change to a caching scheme.

two more points

Finally, two more points to add.

The first point is the previous "When Synchronized encounters this thing, there is a big hole, pay attention! 》 this article, there is a wrong place.

The framed area was added later by me.

After the article was published last week, about a dozen readers gave me feedback on this issue.

I'm really happy, because someone really took my sample code and ran it, and thought about it carefully, and then came to discuss it with me and help me correct what I wrote wrong.

Let me share with you my article "When I read technical articles, what do I think?

It expresses my attitude towards watching technical blogs:

Think one step further when reading technical articles, and sometimes you will have a deeper understanding.

went to the blog with skepticism and falsified with the idea of proof.

Think more about why, there will always be gains.

The second point is this.

I actually wrote an article about computeIfAbsent of ConcurrentHashMap: "Shock! There is also an infinite loop in ConcurrentHashMap, what about the "Easter Eggs" left by the author?

Old readers should have read this article.

When I was wandering on the seata official website before, I saw this blog:

https://seata.io/zh-cn/blog/seata-dsproxy-deadlock.html

The name is "Seata Deadlock Problem Caused by ConcurrentHashMap", I just clicked into it and took a look:

The article mentioned here was written by me.

It is an amazing experience to come across your own article on the seata official website.

Rounding up, I can be considered a man who has contributed to seata.

And you can see that this article actually mentions many articles I have written before. All this knowledge is connected by a small point, from point to line, from line to surface, which is why I insist on writing.

Let's share.

Finally, echoing the beginning of the article, the postgraduate entrance examination is about to check the scores. I know that many of my readers are still taking the postgraduate entrance examination this year.

If you see this, then the following picture is for you:

This article has been included in the personal blog, more original good articles, welcome everyone to play:

https://www.whywhy.vip/