Principle and Implementation of Adaptive Load Balancing Algorithm

background

When choosing a load balancing algorithm, we hope to meet the following requirements:

1. With the compatibility of partition and computer room scheduling

   • The node selected each time is as low as possible
   • Choose the node with the fastest response every time possible

2. No manual intervention on the failed node

   • When a node fails, the load balancing algorithm can automatically isolate the node
   • When the failed node recovers, the traffic distribution to the node can be automatically restored

Based on these considerations, go-zero chose the p2c+EWMA algorithm to implement.

The core idea of the algorithm

p2c

p2c (Pick Of 2 Choices) Choose one of two: Randomly select two nodes from multiple nodes.

go-zero will randomly select 3 times. If the health condition of the node selected at one time meets the requirements, the selection will be interrupted and these two nodes will be used.

EWMA

EWMA (Exponentially Weighted Moving-Average) Exponential moving weighted average method: It means that the weighting coefficient of each value decreases exponentially with time. The closer the value is to the current moment, the larger the weighting coefficient, which reflects the average value in the most recent period of time.

• formula:

  

• Variable explanation:

  • Vt : Represents the EWMA value t
  • Vt-1 : represents the first t-1 request times EWMA value
  • β : is a constant

Advantages of the EWMA algorithm

1. Compared with the ordinary average calculation algorithm, EWMA does not need to save all the past values, the calculation amount is significantly reduced, and the storage resources are also reduced.

2. The traditional average calculation algorithm is not sensitive to network time consumption, but EWMA can adjust β by frequently requesting, and then quickly monitor the network glitch or more to reflect the overall average.

   • When the request is more frequent, it means that the network load of the node has increased. We want to monitor the time β . β smaller the EWMA value is to this time-consuming, and the network glitch can be detected quickly;
   • When the request is relatively infrequent, we relatively value of 1611ccc203c477 β. EWMA value calculated in this way is closer to the average

Beta calculation

go-zero uses Newton's law of cooling decay function calculation model EWMA algorithm β values:

Where Δt is the interval between two requests, e and k are constants

Implement a custom load balancer in gRPC

1. First of all, we need to implement the google.golang.org/grpc/balancer/base/base.go/PickerBuilder Build method in the interface when the service node is updated

   type PickerBuilder interface {
       // Build returns a picker that will be used by gRPC to pick a SubConn.
       Build(info PickerBuildInfo) balancer.Picker
   }

2. Also implement the google.golang.org/grpc/balancer/balancer.go/Picker interface. This interface mainly implements load balancing and selects a node for request use

   type Picker interface {
     Pick(info PickInfo) (PickResult, error)
   }

3. Finally, register the load balancer we implemented with load balance map

The main logic of go-zero to achieve load balancing

1. In each node update, gRPC will call the Build method, at this time, all node information is saved Build
2. gRPC gets the node processing request, it will call the Pick method to get the node. go-zero in Pick method was achieved p2c algorithm, the selection of nodes by the node calculation load EWMA value, returns the low load node for gRPC use.
3. At the end of the request, gRPC will call the PickResult.Done method. go-zero realizes the storage of the time-consuming information of this request, and calculates the EWMA value and saves it for use in the calculation of load in the next request. .

Load balancing code analysis

1. Save all node information of the service

   EWMA by the node to process this request, 0611ccc203ca97 and other information. go-zero designed the following structure for each node:

   type subConn struct {
       addr     resolver.Address
       conn     balancer.SubConn
       lag      uint64 // 用来保存 ewma 值
       inflight int64  // 用在保存当前节点正在处理的请求总数
       success  uint64 // 用来标识一段时间内此连接的健康状态
       requests int64  // 用来保存请求总数
       last     int64  // 用来保存上一次请求耗时, 用于计算 ewma 值
       pick     int64  // 保存上一次被选中的时间点
   }

2. p2cPicker implements the balancer.Picker interface, and conns saves all the node information of the service

   type p2cPicker struct {
     conns []*subConn  // 保存所有节点的信息 
     r     *rand.Rand
     stamp *syncx.AtomicDuration
     lock  sync.Mutex
   }

3. gRPC Build method when the node is updated, and pass in all the node information. Here we save each node information with the subConn structure. And merge them together and save them p2cPicker

   func (b *p2cPickerBuilder) Build(info base.PickerBuildInfo) balancer.Picker {
     ......
     var conns []*subConn
     for conn, connInfo := range readySCs {
       conns = append(conns, &subConn{
         addr:    connInfo.Address,
         conn:    conn,
         success: initSuccess,
       })
     }
     return &p2cPicker{
       conns: conns,
       r:     rand.New(rand.NewSource(time.Now().UnixNano())),
       stamp: syncx.NewAtomicDuration(),
     }
   }

4. Randomly select node information, there are three cases here:

   1. There is only one service node, at this time, it can be gRPC
   2. There are two service nodes, the EWMA value, and the node with low load is returned for use gRPC
   3. There are multiple service nodes. At this time, p2c algorithm, the load conditions are compared, and the node with low load is returned for use gRPC

   The main implementation code is as follows:

   switch len(p.conns) {
     case 0:// 没有节点，返回错误
       return emptyPickResult, balancer.ErrNoSubConnAvailable
     case 1:// 有一个节点，直接返回这个节点
       chosen = p.choose(p.conns[0], nil)
     case 2:// 有两个节点，计算负载，返回负载低的节点
       chosen = p.choose(p.conns[0], p.conns[1])
     default:// 有多个节点，p2c 挑选两个节点，比较这两个节点的负载，返回负载低的节点
       var node1, node2 *subConn
           // 3次随机选择两个节点
       for i := 0; i < pickTimes; i++ {
         a := p.r.Intn(len(p.conns))
         b := p.r.Intn(len(p.conns) - 1)
         if b >= a {
           b++
         }
         node1 = p.conns[a]
         node2 = p.conns[b]
         // 如果这次选择的节点达到了健康要求, 就中断选择
         if node1.healthy() && node2.healthy() {
           break
         }
       }
       // 比较两个节点的负载情况，选择负载低的
       chosen = p.choose(node1, node2)
     }

5. load compute node load situation

   The above choose method will call the load method to calculate the node load.

   The formula for calculating the load is: load = ewma * inflight

   Here is a brief explanation: ewma equivalent to the average request time, inflight is the number of requests being processed by the current node, and the multiplication roughly calculates the network load of the current node.

   func (c *subConn) load() int64 {
     // 通过 EWMA 计算节点的负载情况； 加 1 是为了避免为 0 的情况
     lag := int64(math.Sqrt(float64(atomic.LoadUint64(&c.lag) + 1)))
     load := lag * (atomic.LoadInt64(&c.inflight) + 1)
     if load == 0 {
       return penalty
     }
     return load
   }

6. The request is over, update the node's EWMA and other information

   1. Decrease the total number of requests being processed by the node by 1
   2. Saving processing time point of end of the request, for calculating a difference value from the previous processing request node, and calculates EWMA in beta] value
   3. Calculate the time-consuming of this request, and calculate the EWMA value and save it to the node's lag attribute

   4. The health status of the computing node is saved to the node's success attribute

      func (p *p2cPicker) buildDoneFunc(c *subConn) func(info balancer.DoneInfo) {
      start := int64(timex.Now())
      return func(info balancer.DoneInfo) {
       // 正在处理的请求数减 1
       atomic.AddInt64(&c.inflight, -1)
       now := timex.Now()
       // 保存本次请求结束时的时间点，并取出上次请求时的时间点
       last := atomic.SwapInt64(&c.last, int64(now))
       td := int64(now) - last
       if td < 0 {
         td = 0
       }
       // 用牛顿冷却定律中的衰减函数模型计算EWMA算法中的β值
       w := math.Exp(float64(-td) / float64(decayTime))
       // 保存本次请求的耗时
       lag := int64(now) - start
       if lag < 0 {
         lag = 0
       }
       olag := atomic.LoadUint64(&c.lag)
       if olag == 0 {
         w = 0
       }
       // 计算 EWMA 值
       atomic.StoreUint64(&c.lag, uint64(float64(olag)*w+float64(lag)*(1-w)))
       success := initSuccess
       if info.Err != nil && !codes.Acceptable(info.Err) {
         success = 0
       }
       osucc := atomic.LoadUint64(&c.success)
       atomic.StoreUint64(&c.success, uint64(float64(osucc)*w+float64(success)*(1-w)))
      
       stamp := p.stamp.Load()
       if now-stamp >= logInterval {
         if p.stamp.CompareAndSwap(stamp, now) {
           p.logStats()
         }
       }
      }
      }

project address

https://github.com/tal-tech/go-zero

Welcome to use go-zero and star support us!

WeChat Exchange Group

Follow the " Practice " public account and click on the exchange group get the community group QR code.