This article explains the principle and implementation of consistent hashing

Why a consistent hash is needed

First introduce what is a hash

Hash, generally translated as a hash, or transliterated as a hash, is to transform an input of any length (also called a pre-mapped pre-image) into a fixed-length output through a hashing algorithm, and the output is the hash value. This conversion is a compression mapping, that is, the hash value space is usually much smaller than the input space, and different inputs may be hashed into the same output, so it is impossible to determine the unique input value from the hash value. Simply put, it is a function that compresses messages of any length to a fixed-length message digest.

In a distributed cache service, it is often necessary to perform node addition and deletion operations on the service. What we hope is that the node addition and deletion operations minimize the update of the data-node mapping relationship.

If we use the hash modulus (hash(key)%nodes) algorithm as a routing strategy:

The disadvantage of hash modulus is that if there are node deletion and addition operations, the effect on the result of hash(key)%nodes is too large, causing a large number of requests to fail to hit and causing cached data to be reloaded.

Based on the above shortcomings, a new algorithm is proposed: consistent hashing. Consistent hashing can realize that node deletion and addition will only affect the mapping relationship of a small part of the data. Because of this feature, the hash algorithm is also often used in various equalizers to achieve smooth migration of system traffic.

How consistent hashing works

First, perform a hash calculation on the node, and the hash value is usually in the range of 2^32-1. Then we abstract the end-to-end connection of the interval 2^32-1 into a ring and map the hash value of the node to the ring. When we want to query the target node of the key, we also hash the key and then clockwise The first node found is the target node.

According to the principle, we analyze the impact of node addition and deletion on the data range.

1. Node add

   

   Only the data between the new node and the previous node (the first node that the new node is searched counterclockwise) will be affected.

2. Node deletion

   

   Only the data between the deleted node and the previous node (the first node that the deleted node looks up counterclockwise) will be affected.

Is that all done? Not yet, just imagine if the number of nodes on the ring is very small, then it is very likely that the data distribution will be unbalanced. In essence, the granularity of the interval distribution on the ring is too coarse.

How to solve it? Isn't the granularity too coarse? Then add more nodes, which leads to the concept of virtual nodes of consistent hashing. The function of virtual nodes is to make the distribution of nodes on the ring finer.

One real node corresponds to multiple virtual nodes, and the hash value of the virtual node is mapped to the ring. To query the target node of the key, we first query the virtual node and then find the real node.

Code

Based on the above principle of consistent hashing, we can extract the core functions of consistent hashing:

1. Add node
2. Delete node
3. Query node

Let's define the interface:

ConsistentHash interface {
    Add(node Node)
    Get(key Node) Node
    Remove(node Node)
}

In reality, the service capabilities of different nodes may be different due to hardware differences, so we hope to specify the weight when adding nodes. The so-called weight in consistent hashing means that we want the target node of the key to hit the probability ratio. A real node with a large number of virtual nodes means a high probability of being hit.

In the interface definition, we can add two methods: support adding nodes by specifying the number of virtual nodes, and adding nodes by weight. In essence, it will eventually reflect the difference in the probability distribution caused by the difference in the number of virtual nodes.

When specifying the weight: the number of actual virtual nodes = configured virtual nodes * weight/100

ConsistentHash interface {
    Add(node Node)
    AddWithReplicas(node Node, replicas int)
    AddWithWeight(node Node, weight int)
    Get(key Node) Node
    Remove(node Node)
}

Next, consider several project implementation issues:

1. How are virtual nodes stored?

   It's very simple, just use a list (slice) to store it.

2. Virtual node-real node relational storage

   map is fine.

3. Query clockwise how the first virtual node is implemented

   To keep the list of virtual nodes in order, search for the first index larger than hash(key) in binary, just list[index].

4. There is a small probability of conflicts when virtual nodes are hashed. How to deal with it?

   Conflict means that this virtual node corresponds to multiple real nodes. The value in the map stores an array of real nodes, and the nodes are modulo when querying the target node of the key.

5. How to generate virtual nodes

   Configure the replicas based on the number of virtual nodes, and add i bytes to the replicas in sequence for hash calculation.

go-zero source code analysis

core/hash/consistenthash.go

Detailed notes can be viewed: https://github.com/Ouyangan/go-zero-annotation/blob/84ae351e4ebce558e082d54f4605acf750f5d285/core/hash/consistenthash.go

It took a day to read the go-zero source code consistency hash source code. The writing is really good, and all the details have been considered.

The hash function used by go-zero is MurmurHash3 , GitHub: https://github.com/spaolacci/murmur3

go-zero does not define the interface, it doesn't matter, just look at the structure ConsistentHash :

// Func defines the hash method.
// 哈希函数
Func func(data []byte) uint64

// A ConsistentHash is a ring hash implementation.
// 一致性哈希
ConsistentHash struct {
    // 哈希函数
    hashFunc Func
    // 确定node的虚拟节点数量
    replicas int
    // 虚拟节点列表
    keys []uint64
    // 虚拟节点到物理节点的映射
    ring map[uint64][]interface{}
    // 物理节点映射，快速判断是否存在node
    nodes map[string]lang.PlaceholderType
    // 读写锁
    lock sync.RWMutex
}

Hash calculation of key and virtual node

Before hashing, you must first convert the key to a string

// 可以理解为确定node字符串值的序列化方法
// 在遇到哈希冲突时需要重新对key进行哈希计算
// 为了减少冲突的概率前面追加了一个质数prime来减小冲突的概率
func innerRepr(v interface{}) string {
   return fmt.Sprintf("%d:%v", prime, v)
}

// 可以理解为确定node字符串值的序列化方法
// 如果让node强制实现String()会不会更好一些？
func repr(node interface{}) string {
   return mapping.Repr(node)
}

Here mapping.Repr will determine the fmt.Stringer interface, if it meets, it will call its String method. go-zero code for 061a58cbd773dc is as follows:

// Repr returns the string representation of v.
func Repr(v interface{}) string {
    if v == nil {
        return ""
    }

    // if func (v *Type) String() string, we can't use Elem()
    switch vt := v.(type) {
    case fmt.Stringer:
        return vt.String()
    }

    val := reflect.ValueOf(v)
    if val.Kind() == reflect.Ptr && !val.IsNil() {
        val = val.Elem()
    }

    return reprOfValue(val)
}

Add node

The final call is to specify the virtual node to add the node method

// 扩容操作，增加物理节点
func (h *ConsistentHash) Add(node interface{}) {
    h.AddWithReplicas(node, h.replicas)
}

Add node-specify weight

The final call is also to specify the virtual node to add the node method

// 按权重添加节点
// 通过权重来计算方法因子，最终控制虚拟节点的数量
// 权重越高，虚拟节点数量越多
func (h *ConsistentHash) AddWithWeight(node interface{}, weight int) {
    replicas := h.replicas * weight / TopWeight
    h.AddWithReplicas(node, replicas)
}

Add node-specify the number of virtual nodes

// 扩容操作，增加物理节点
func (h *ConsistentHash) AddWithReplicas(node interface{}, replicas int) {
    // 支持可重复添加
    // 先执行删除操作
    h.Remove(node)
    // 不能超过放大因子上限
    if replicas > h.replicas {
        replicas = h.replicas
    }
    // node key
    nodeRepr := repr(node)
    h.lock.Lock()
    defer h.lock.Unlock()
    // 添加node map映射
    h.addNode(nodeRepr)
    for i := 0; i < replicas; i++ {
        // 创建虚拟节点
        hash := h.hashFunc([]byte(nodeRepr + strconv.Itoa(i)))
        // 添加虚拟节点
        h.keys = append(h.keys, hash)
        // 映射虚拟节点-真实节点
        // 注意hashFunc可能会出现哈希冲突，所以采用的是追加操作
        // 虚拟节点-真实节点的映射对应的其实是个数组
        // 一个虚拟节点可能对应多个真实节点，当然概率非常小
        h.ring[hash] = append(h.ring[hash], node)
    }
    // 排序
    // 后面会使用二分查找虚拟节点
    sort.Slice(h.keys, func(i, j int) bool {
        return h.keys[i] < h.keys[j]
    })
}

Delete node

// 删除物理节点
func (h *ConsistentHash) Remove(node interface{}) {
    // 节点的string
    nodeRepr := repr(node)
    // 并发安全
    h.lock.Lock()
    defer h.lock.Unlock()
    // 节点不存在
    if !h.containsNode(nodeRepr) {
        return
    }
    // 移除虚拟节点映射
    for i := 0; i < h.replicas; i++ {
        // 计算哈希值
        hash := h.hashFunc([]byte(nodeRepr + strconv.Itoa(i)))
        // 二分查找到第一个虚拟节点
        index := sort.Search(len(h.keys), func(i int) bool {
            return h.keys[i] >= hash
        })
        // 切片删除对应的元素
        if index < len(h.keys) && h.keys[index] == hash {
            // 定位到切片index之前的元素
            // 将index之后的元素（index+1）前移覆盖index
            h.keys = append(h.keys[:index], h.keys[index+1:]...)
        }
        // 虚拟节点删除映射
        h.removeRingNode(hash, nodeRepr)
    }
    // 删除真实节点
    h.removeNode(nodeRepr)
}

// 删除虚拟-真实节点映射关系
// hash - 虚拟节点
// nodeRepr - 真实节点
func (h *ConsistentHash) removeRingNode(hash uint64, nodeRepr string) {
    // map使用时应该校验一下
    if nodes, ok := h.ring[hash]; ok {
        // 新建一个空的切片,容量与nodes保持一致
        newNodes := nodes[:0]
        // 遍历nodes
        for _, x := range nodes {
            // 如果序列化值不相同，x是其他节点
            // 不能删除
            if repr(x) != nodeRepr {
                newNodes = append(newNodes, x)
            }
        }
        // 剩余节点不为空则重新绑定映射关系
        if len(newNodes) > 0 {
            h.ring[hash] = newNodes
        } else {
            // 否则删除即可
            delete(h.ring, hash)
        }
    }
}

Query node

// 根据v顺时针找到最近的虚拟节点
// 再通过虚拟节点映射找到真实节点
func (h *ConsistentHash) Get(v interface{}) (interface{}, bool) {
    h.lock.RLock()
    defer h.lock.RUnlock()
    // 当前没有物理节点
    if len(h.ring) == 0 {
        return nil, false
    }
    // 计算哈希值
    hash := h.hashFunc([]byte(repr(v)))
    // 二分查找
    // 因为每次添加节点后虚拟节点都会重新排序
    // 所以查询到的第一个节点就是我们的目标节点
    // 取余则可以实现环形列表效果，顺时针查找节点
    index := sort.Search(len(h.keys), func(i int) bool {
        return h.keys[i] >= hash
    }) % len(h.keys)
    // 虚拟节点->物理节点映射
    nodes := h.ring[h.keys[index]]
    switch len(nodes) {
    // 不存在真实节点
    case 0:
        return nil, false
    // 只有一个真实节点，直接返回
    case 1:
        return nodes[0], true
    // 存在多个真实节点意味这出现哈希冲突
    default:
        // 此时我们对v重新进行哈希计算
        // 对nodes长度取余得到一个新的index
        innerIndex := h.hashFunc([]byte(innerRepr(v)))
        pos := int(innerIndex % uint64(len(nodes)))
        return nodes[pos], true
    }
}

project address

https://github.com/zeromicro/go-zero

Welcome to use go-zero and star support us!

WeChat Exchange Group

Follow the " Practice " public account and click on the exchange group get the community group QR code.