go-zero microservice combat series (8. How to handle tens of thousands of order requests per second)

In the previous articles, we spent a lot of time introducing how to use the cache to optimize the read performance of the system. The reason is that most of our products are a scenario of more reading and less writing, especially in the early stage of the product, it may be Most users just come to check the products, and very few users actually place an order. However, with the development of the business, we will encounter some scenarios of high concurrent write requests. The most typical high concurrent write scenario is the flash purchase. After the flash-buying starts, users will frantically refresh the page so that they can see the products as soon as possible, so the flash-kill scene is also a high-concurrency reading scene. So how do we optimize for high concurrent read and write scenarios?

Process hot data

The data of the spike is usually hot data, and there are generally several ideas for processing hot data: one is optimization, the other is restriction, and the third is isolation.

optimization

The most effective way to optimize hotspot data is to cache hotspot data. We can cache hotspot data in memory cache.

limit

The restriction is more of a protection mechanism. When the seckill starts, the user will continuously refresh the page to obtain data. At this time, we can limit the number of requests for a single user, such as only one request per second, and an error will be returned directly if the limit is exceeded. The returned error should be as user-friendly as possible, such as a friendly prompt such as "The shop assistant is busy".

isolation

The first principle of the seckill system design is to isolate this kind of hot data, so as not to let 1% of the requests affect the other 99%, and it is more convenient to optimize the 1% of the requests after isolation. In terms of implementation, we need to do service isolation, that is, the seckill function is an independent service, and the notification needs to be isolated from data. Most of the seckill calls are hot data, we need to use a separate Redis cluster and a separate Mysql, the purpose is not to want Let 1% of data have a chance to affect 99% of data.

Traffic clipping

For the seckill scenario, it is characterized by the influx of a large number of requests at the moment when the seckill starts, which will lead to a particularly high traffic peak. But in the end, the number of people who can grab the product is fixed, that is, whether it is 100 people or 10,000,000 people, the result of the request is the same. The higher the concurrency, the more invalid requests. But from a business point of view, the seckill event hopes that more people will participate, that is, when the seckill starts, more people are expected to refresh the page, but when the order is actually placed, the more requests, the better. . Therefore, we can design some rules to delay concurrent requests more, and even filter out some invalid requests.

In essence, peak shaving is to delay the issuance of user requests more in order to reduce and filter out some invalid requests. It follows the principle that the number of requests should be as small as possible. The easiest solution we can think of is to use message queues to buffer instantaneous traffic, convert synchronous direct calls into asynchronous indirect pushes, and use a queue in the middle to undertake instantaneous traffic peaks at one end, and push messages smoothly at the other end. ,As shown below:

After using the message queue for asynchronous processing, the result of the seckill is not easy to be returned synchronously, so our idea is that when the user initiates the seckill request, the prompt message that responds to the user "Seckill result is being calculated..." will be returned synchronously. How do we return the result to the user after the calculation is done? In fact, there are many options.

• One is to use the polling method in the page to regularly and actively go to the server to query the results. For example, request the server every second to see if there is a processing result. The disadvantage of this method is that the number of requests on the server will increase a lot.
• The second is the method of active push, which requires the server and the client to maintain a long connection. After the server processes the request, it actively pushes it to the client. The disadvantage of this method is that the number of connections on the server will be larger.

Another question is what if the asynchronous request fails? I think for the instant kill scenario, it’s better to just throw it away if it fails. The worst result is that the user doesn’t grab it. If you want to ensure fairness as much as possible, you can retry after failure.

How to ensure that messages are only consumed once

Kafka can guarantee the mechanism of "At Least Once", that is, the message will not be lost, but it may lead to repeated consumption. Once the message is repeatedly consumed, it will cause errors in business logic processing, so how can we avoid repeated consumption of messages Woolen cloth?

We only need to ensure that even if repeated messages are consumed, the final result of consumption is equivalent to the result of only one consumption, that is, to ensure that the process of message production and consumption is idempotent. What is idempotent? If we want to reduce the existing inventory quantity by 1 when we consume a message, then if we consume two identical messages, the inventory quantity is reduced by 2, which is not idempotent. And if the processing logic after consuming a message is to set the inventory quantity to 0, or subtract 1 if the current inventory quantity is 10, the result obtained when consuming multiple messages is the same, which is idempotent of. To put it bluntly, no matter how many times you do something and once you do it, the result is the same, then this is idempotency.

We can store the unique id in the database after the message is consumed. The unique id here can use the combination of the user id and the commodity id. Before processing the next message, first query the id from the database to see if it has been consumed. If Give up after spending. The pseudo code is as follows:

 isConsume := getByID(id)
if isConsume {
  return  
} 
process(message)
save(id)

Another way is to ensure idempotency through a unique index in the database, but this depends on the specific business and will not be repeated here.

Code

The entire seckill flow chart is as follows:

I use kafka as a message queue, so I need to install kafka locally first. I use a mac that can be installed directly with homebrew, and kafka's dependency on zookeeper will also be installed automatically

 brew install kafka

After installation, start zookeeper and kafka through brew services start, kafka listens on port 9092 by default

 brew services start zookeeper

brew services start kafka

The SeckillOrder method of seckill-rpc implements the seckill logic. We first limit the number of user requests, such as limiting users to only one request per second. Here, the PeriodLimit function provided by go-zero is used to implement it. If the limit is exceeded, it will return directly

 code, _ := l.limiter.Take(strconv.FormatInt(in.UserId, 10))
if code == limit.OverQuota {
  return nil, status.Errorf(codes.OutOfRange, "Number of requests exceeded the limit")
}

Then check the inventory of the current snapped-up products. If the inventory is insufficient, return it directly. If the inventory is sufficient, you can enter the order process and send a message to kafka. Here kafka uses the kq library provided by go-zero, which is very simple and easy to use. Create a new topic and configure the initialization and logic as follows:

 Kafka:
  Addrs:
    - 127.0.0.1:9092
  SeckillTopic: seckill-topic

 KafkaPusher: kq.NewPusher(c.Kafka.Addrs, c.Kafka.SeckillTopic)

 p, err := l.svcCtx.ProductRPC.Product(l.ctx, &product.ProductItemRequest{ProductId: in.ProductId})
if err != nil {
  return nil, err
}
if p.Stock <= 0 {
  return nil, status.Errorf(codes.OutOfRange, "Insufficient stock")
}
kd, err := json.Marshal(&KafkaData{Uid: in.UserId, Pid: in.ProductId})
if err != nil {
  return nil, err
}
if err := l.svcCtx.KafkaPusher.Push(string(kd)); err != nil {
  return nil, err
}

seckill-rmq consumes the data produced by seckill-rpc to place an order. We create a new seckill-rmq service with the following structure:

 tree ./rmq

./rmq
├── etc
│   └── seckill.yaml
├── internal
│   ├── config
│   │   └── config.go
│   └── service
│       └── service.go
└── seckill.go

4 directories, 4 files

Still use kq to initialize the startup service, here we need to register a ConsumeHand method, which is used to consume kafka data

 srv := service.NewService(c)
queue := kq.MustNewQueue(c.Kafka, kq.WithHandle(srv.Consume))
defer queue.Stop()

fmt.Println("seckill started!!!")
queue.Start()

In the Consume method, after consuming the data, first deserialize it, and then call product-rpc to view the inventory of the current product. If the inventory is sufficient, we think we can place an order, call order-rpc to create an order, and finally update the inventory.

 func (s *Service) Consume(_ string, value string) error {
  logx.Infof("Consume value: %s\n", value)
  var data KafkaData
  if err := json.Unmarshal([]byte(value), &data); err != nil {
    return err
  }
  p, err := s.ProductRPC.Product(context.Background(), &product.ProductItemRequest{ProductId: data.Pid})
  if err != nil {
    return err
  }
  if p.Stock <= 0 {
    return nil
  }
  _, err = s.OrderRPC.CreateOrder(context.Background(), &order.CreateOrderRequest{Uid: data.Uid, Pid: data.Pid})
  if err != nil {
    logx.Errorf("CreateOrder uid: %d pid: %d error: %v", data.Uid, data.Pid, err)
    return err
  }
  _, err = s.ProductRPC.UpdateProductStock(context.Background(), &product.UpdateProductStockRequest{ProductId: data.Pid, Num: 1})
  if err != nil {
    logx.Errorf("UpdateProductStock uid: %d pid: %d error: %v", data.Uid, data.Pid, err)
    return err
  }
  // TODO notify user of successful order placement
  return nil
}

Two tables, orders and orderitem, are involved in the process of creating an order, so we need to use local transactions for insertion. The code is as follows:

 func (m *customOrdersModel) CreateOrder(ctx context.Context, oid string, uid, pid int64) error {
  _, err := m.ExecCtx(ctx, func(ctx context.Context, conn sqlx.SqlConn) (sql.Result, error) {
    err := conn.TransactCtx(ctx, func(ctx context.Context, session sqlx.Session) error {
      _, err := session.ExecCtx(ctx, "INSERT INTO orders(id, userid) VALUES(?,?)", oid, uid)
      if err != nil {
        return err
      }
      _, err = session.ExecCtx(ctx, "INSERT INTO orderitem(orderid, userid, proid) VALUES(?,?,?)", "", uid, pid)
      return err
    })
    return nil, err
  })
  return err
}

The order number generation logic is as follows. Here, the time plus the auto-increment number is used to generate the order.

 var num int64

func genOrderID(t time.Time) string {
  s := t.Format("20060102150405")
  m := t.UnixNano()/1e6 - t.UnixNano()/1e9*1e3
  ms := sup(m, 3)
  p := os.Getpid() % 1000
  ps := sup(int64(p), 3)
  i := atomic.AddInt64(&num, 1)
  r := i % 10000
  rs := sup(r, 4)
  n := fmt.Sprintf("%s%s%s%s", s, ms, ps, rs)
  return n
}

func sup(i int64, n int) string {
  m := fmt.Sprintf("%d", i)
  for len(m) < n {
    m = fmt.Sprintf("0%s", m)
  }
  return m
}

Finally, start the product-rpc, order-rpc, seckill-rpc and seckill-rmq services, as well as zookeeper, kafka, mysql and redis. After starting, we call seckill-rpc to place orders in seconds

 grpcurl -plaintext -d '{"user_id": 111, "product_id": 10}' 127.0.0.1:9889 seckill.Seckill.SeckillOrder

The consumption record is printed in seckill-rmq, the output is as follows

 {"@timestamp":"2022-06-26T10:11:42.997+08:00","caller":"service/service.go:35","content":"Consume value: {\"uid\":111,\"pid\":10}\n","level":"info"}

At this time, check that the order has been created in the orders table, and the inventory of the product is reduced by one

concluding remarks

In essence, seckill is a scenario of high concurrent reading and high concurrent writing. Above we introduced the precautions and optimization points of seckill. Our seckill scenario is relatively simple, but in fact there is no general seckill framework. We need To optimize according to the actual business scenario, the means of request optimization of different magnitudes are also different. Here we only show the relevant optimization of the server, but for the second-kill scenario, the entire request link needs to be optimized. For example, for static data, we can use CDN for acceleration. In order to prevent traffic floods, we can set the answering function on the front end, etc. .

Hope this article is helpful to you, thank you.

Updated every Monday and Thursday

Code repository: https://github.com/zhoushuguang/lebron

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.