In-process caching helps you improve concurrency!

Preface

Cache, the original intention of the design is to reduce heavy IO operations and increase system concurrency. Whether it is the CPU multi-level cache, page cache , or the familiar redis cache in our business, the essence is to store limited hotspot data in a storage medium with faster access.

The cache design of the computer itself is that the CPU adopts a multi-level cache. So for our services, can we also use this multi-level caching approach to organize our cached data? At the same time, redis will be accessed through the network IO, so can we directly store the hot data in this process, and the process itself will cache a copy of the most recent batch of data?

This leads to what we are discussing today: local cache , local cache, also called process cache.

This article takes you to discuss the design of the process cache in go-zero Let's go!

Quick start

As a process storage design, of course crud has:

1. We first initialize local cache

// 先初始化 local cache
cache, err = collection.NewCache(time.Minute, collection.WithLimit(10))
if err != nil {
  log.Fatal(err)
}

The meaning of the parameters:

• expire : unified key expiration time
• CacheOption : cache settings. For example, the upper limit setting of the key, etc.

1. Basic operation cache

// 1. add/update 增加/修改都是该API
cache.Set("first", "first element")

// 2. get 获取key下的value
value, ok := cache.Get("first")

// 3. del 删除一个key
cache.Del("first")

• Set(key, value) set cache
• value, ok := Get(key) read cache
• Del(key) delete cache

1. Advanced operation

cache.Take("first", func() (interface{}, error) {
  // 模拟逻辑写入local cache
  time.Sleep(time.Millisecond * 100)
  return "first element", nil
})

The previous Set(key, value) simply adds <key, value> to the cache; Take(key, setFunc) fetch method when the value for the key does not exist, gives the specific reading logic to the developer for implementation, and automatically puts the result in the cache.

At this point, the core usage code is basically finished, but it actually looks quite simple. You can also go to https://github.com/tal-tech/go-zero/blob/master/core/collection/cache_test.go to see the usage in test.

solution

First of all, cache is essentially a medium that stores limited hot data. It faces the following problems:

1. Limited capacity
2. Hot data statistics
3. Multithreaded access

Let's talk about our design practice in these three areas.

Limited capacity

Limited means we must eliminate when it is full, and this involves elimination strategies. cache uses: LRU (least used recently).

How does the elimination happen? There are several options for

1. Start a timer, continue to cycle all keys, wait until the preset expiration time, execute the callback function (here is to delete the keys in the map)
2. Lazy deletion. Judge whether the key is deleted during access. The disadvantage is: if it is not visited, it will increase the waste of space.

The cache taken in is first automatically deleted . However, the biggest problems encountered in active deletion are:

continuously loops and consumes CPU resources. Even if it is done in an additional coroutine, it is not necessary.

cache , a time wheel is used to record additional overdue notifications. channel there is a notification in 060ab01a2be164 overdue, then the delete callback is triggered.

More design articles about time wheel https://go-zero.dev/cn/timing-wheel.html

Hot data statistics

For the cache, we need to know whether the cache is valuable when using additional space and code, and we want to know whether we need to further optimize the expiration time or the cache size, all of which we rely on statistical capabilities, go-zero sqlc and mongoc also provide statistical capabilities. Therefore, the cache that we cache provides developers with the feature of local cache monitoring. When accessing ELK , developers can more intuitively monitor the distribution of the cache.

The design is actually very simple, that is: Get() hits, just add 1 to the count to .

func (c *Cache) Get(key string) (interface{}, bool) {
  value, ok := c.doGet(key)
  if ok {
    // 命中hit+1
    c.stats.IncrementHit()
  } else {
    // 未命中miss+1
    c.stats.IncrementMiss()
  }

  return value, ok
}

Multithreaded access

When multiple coroutines are accessed concurrently, the following issues are involved for the cache:

• Write-write conflict
• LRU element movement process conflict
• Concurrent execution of write cache, causing traffic shock or invalid traffic

In this case, write better solve the conflict, the easiest way is to lock :

// Set(key, value)
func (c *Cache) Set(key string, value interface{}) {
  // 加锁，然后将 <key, value> 作为键值对写入 cache 中的 map
  c.lock.Lock()
  _, ok := c.data[key]
  c.data[key] = value
  // lru add key
  c.lruCache.add(key)
  c.lock.Unlock()
  ...
}

// 还有一个在操作 LRU 的地方时：Get()
func (c *Cache) doGet(key string) (interface{}, bool) {
  c.lock.Lock()
  defer c.lock.Unlock()
  // 当key存在时，则调整 LRU item 中的位置，这个过程也是加锁的
  value, ok := c.data[key]
  if ok {
    c.lruCache.add(key)
  }

  return value, ok
}

The concurrent execution of writing logic is mainly passed in by the developer himself. And this process:

func (c *Cache) Take(key string, fetch func() (interface{}, error)) (interface{}, error) {
  // 1. 先获取 doGet() 中的值
  if val, ok := c.doGet(key); ok {
    c.stats.IncrementHit()
    return val, nil
  }

  var fresh bool
  // 2. 多协程中通过 sharedCalls 去获取，一个协程获取多个协程共享结果
  val, err := c.barrier.Do(key, func() (interface{}, error) {
    // double check，防止多次读取
    if val, ok := c.doGet(key); ok {
      return val, nil
    }
    ...
    // 重点是执行了传入的缓存设置函数
    val, err := fetch()
    ...
    c.Set(key, val)
  })
  if err != nil {
    return nil, err
  }
  ...
  return val, nil
}

However, sharedCalls saves multiple execution functions by sharing the returned result and reduces the coroutine competition.

to sum up

This article explains the practice of local cache design. From use to design ideas, you can also dynamically modify the cache expiration strategy according to your business, add the statistical indicator you want to implement your own local cache.

You can even combine local caching with redis to provide multi-level caching for the service. This is left to our next article: , a multi-level design for caching in the service .

go-zero more design and implementation articles about 060ab01a2be453, you can follow the "Microservice Practice" public account.

project address

https://github.com/tal-tech/go-zero

Welcome to use go-zero and star support us!

WeChat Exchange Group

Follow the " practice " public enter the group obtain the QR code of the community group.

For the go-zero series of articles, please refer to the official account of "Microservice Practice"