Redis application combat-spike scenario (Node.js version)

Write in front

With the increase in business volume, the company has recently spent a few months to fully integrate Redis development process, it was found that there was a lack of specific actual combat information on the market, especially in the Node.js environment. The information that can be found is either too simple to get started. Either the name is not true, most of them belong to the junior level. Therefore, I decided to share the results of the company during this period, and will use several articles to introduce in Redis several usage scenarios of 060d470bed7d57, and I hope everyone can learn and make progress together.
Let's start the first one, the spike scene.

Business analysis

In actual business, the spike contains many scenarios, which can be divided into three stages: before, during and after the spike. The specific analysis from the development perspective is as follows:

1. Before the spike: The main thing is to do a good job of caching to deal with frequent user access, because the data is fixed, you can CDN the elements of the product detail page, and then use 060d470bed7ded or browser to cache.

2. Spike: Mainly inventory inspection, inventory deduction and order processing. The features of this step are

   • In a short period of time, a large number of users are rushing to buy at the same time, the system traffic suddenly increases, and the server pressure increases instantly (instantaneous concurrent access is high)
   • The requested quantity is greater than the product inventory, such as 10,000 users snapped up, but the inventory is only 100
   • Limited users can only purchase within a certain period of time
   • Limit the number of purchases by a single user to avoid buying orders
   • Panic buying is dealing with the database, the core function is to place an order, and the inventory cannot be deducted into a negative number
   • The operation of the database reads more and writes less, and the read operation is relatively simple
3. After the spike: It is mainly for some users to view purchased orders, process refunds and process logistics, etc. At this time, the amount of user requests has dropped, the operation is relatively simple, and the server pressure is not great.

Based on the above analysis, this article focuses on the development and explanation in the spike, and other interested partners can search for the information and try it out by themselves.

Development environment

Database: Redis 3.2.9 + Mysql 5.7.18
Server: Node.js v10.15.0
Test tool: Jmeter-5.4.1

Actual combat

Database preparation

As shown in the figure, three tables need to be created Mysql

• Product table, used to record product information, the fields are Id, name, thumbnail, price and status, etc.
• The spike activity table is used to record the detailed information of the spike activity. The fields are Id, the product Id participating in the spike, inventory, the spike start time, the spike end time, and whether the spike activity is valid, etc.
• Order table, used to record the data after the order is placed, the fields are Id, order number, product Id, purchasing user Id, order status, order type and spike activity Id, etc.

The following is a statement to sql

CREATE TABLE `seckill_goods` (
    `id` INTEGER NOT NULL auto_increment,
    `fk_good_id` INTEGER,
    `amount` INTEGER,
    `start_time` DATETIME,
    `end_time` DATETIME,
    `is_valid` TINYINT ( 1 ),
    `comment` VARCHAR ( 255 ),
    `created_at` DATETIME NOT NULL,
    `updated_at` DATETIME NOT NULL,
PRIMARY KEY ( `id` ) 
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;

CREATE TABLE `orders` (
    `id` INTEGER NOT NULL auto_increment,
    `order_no` VARCHAR ( 255 ),
    `good_id` INTEGER,
    `user_id` INTEGER,
    `status` ENUM ( '-1', '0', '1', '2' ),
    `order_type` ENUM ( '1', '2' ),
    `scekill_id` INTEGER,
    `comment` VARCHAR ( 255 ),
    `created_at` DATETIME NOT NULL,
    `updated_at` DATETIME NOT NULL,
PRIMARY KEY ( `id` ) 
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;

CREATE TABLE `goods` (
    `id` INTEGER NOT NULL auto_increment,
    `name` VARCHAR ( 255 ),
    `thumbnail` VARCHAR ( 255 ),
    `price` INTEGER,
    `status` TINYINT ( 1 ),
    `stock` INTEGER,
    `stock_left` INTEGER,
    `description` VARCHAR ( 255 ),
    `comment` VARCHAR ( 255 ),
    `created_at` DATETIME NOT NULL,
    `updated_at` DATETIME NOT NULL,
PRIMARY KEY ( `id` ) 
) ENGINE = INNODB DEFAULT CHARSET = utf8mb4;

The product table is not the focus of this business. The following logic uses the id=1 product as an example, please be aware.
Create a record with an inventory of 200 in the spike activity table as the spike test data, refer to the following statement:

INSERT INTO `redis_app`.`seckill_goods` (
    `id`,
    `fk_good_id`,
    `amount`,
    `start_time`,
    `end_time`,
    `is_valid`,
    `comment`,
    `created_at`,
    `updated_at` 
)
VALUES
    (
        1,
        1,
        200,
        '2020-06-20 00:00:00',
        '2023-06-20 00:00:00',
        1,
        '...',
        '2020-06-20 00:00:00',
        '2021-06-22 10:18:16' 
    );

Spike interface development

First, talk about the specific development environment in Node.js

• web frame uses Koa2
• mysql operation uses Node.js ORM tools promise Sequelize
• redis operation uses the ioredis library
• The encapsulation ctx.throwException method is used to handle errors, and the encapsulation ctx.send method is used to return the correct result. For specific implementation, refer to the complete code at the end of the article

Secondly, analyze the logic to be processed by the interface. The approximate steps and sequence are as follows:

1. Basic parameter verification
2. Determine whether the product has been snapped up
3. Determine whether the spike activity is effective
4. Determine whether the spike activity has started or ended
5. Determine whether the spike product is sold out
6. Get login user information
7. Determine whether the logged-in user has been grabbed
8. Deduction of inventory
9. Place an order

Finally, based on the analysis, the above steps are initially implemented with code, as follows:

// 引入moment库处理时间相关数据
const moment = require('moment');
// 引入数据库model文件
const seckillModel = require('../../dbs/mysql/models/seckill_goods');
const ordersModel = require('../../dbs/mysql/models/orders');
// 引入工具函数或工具类
const UserModule = require('../modules/user');
const { random_String } = require('../../utils/tools/funcs');

class Seckill {
  /**
   * 秒杀接口
   * 
   * @method post
   * @param good_id 产品id
   * @param accessToken 用户Token
   * @param path 秒杀完成后跳转路径
   */
  async doSeckill(ctx, next) {
    const body = ctx.request.body;
    const accessToken = ctx.query.accessToken;
    const path = body.path;

    // 基本参数校验
    if (!accessToken || !path) { return ctx.throwException(20001, '参数错误！'); };
    // 判断此产品是否加入了抢购
    const seckill = await seckillModel.findOne({
      where: {
        fk_good_id: ctx.params.good_id,
      }
    });
    if (!seckill) { return ctx.throwException(30002, '该产品并未有抢购活动！'); };
    // 判断是否有效
    if (!seckill.is_valid) { return ctx.throwException(30003, '该活动已结束！'); };
    // 判单是否开始、结束
    if(moment().isBefore(moment(seckill.start_time))) {
      return ctx.throwException(30004, '该抢购活动还未开始！');
    }
    if(moment().isAfter(moment(seckill.end_time))) {
      return ctx.throwException(30005, '该抢购活动已经结束！');
    }
    // 判断是否卖完
    if(seckill.amount < 1) { return ctx.throwException(30006, '该产品已经卖完了！'); };

    //获取登录用户信息(这一步只是简单模拟验证用户身份，实际开发中要有严格的accessToken校验流程)
    const userInfo = await UserModule.getUserInfo(accessToken);
    if (!userInfo) { return ctx.throwException(10002, '用户不存在！'); };

    // 判断登录用户是否已抢到（一个用户针对这次活动只能购买一次）
    const orderInfo = await ordersModel.findOne({
      where: {
        user_id: userInfo.id,
        seckill_id: seckill.id,
      },
    });
    if (orderInfo) { return ctx.throwException(30007, '该用户已抢到该产品，无需再抢！'); };

    // 扣库存
    const count = await seckill.decrement('amount');
    if (count.amount <= 0) { return ctx.throwException(30006, '该产品已经卖完了！'); };

    // 下单
    const orderData = {
      order_no: Date.now() + random_String(4), // 这里就用当前时间戳加4位随机数作为订单号，实际开发中根据业务规划逻辑 
      good_id: ctx.params.good_id,
      user_id: userInfo.id,
      status: '1', // -1 已取消, 0 未付款， 1 已付款， 2已退款
      order_type: '2', // 1 常规订单 2 秒杀订单
      seckill_id: seckill.id, // 秒杀活动id
      comment: '', // 备注
    };
    const order = ordersModel.create(orderData);

    if (!order) { return ctx.throwException(30008, '抢购失败!'); };

    ctx.send({
      path,
      data: '抢购成功!'
    });

  }

}

module.exports = new Seckill();

At this point, the spike interface has been implemented using a traditional relational database. The code is not complicated and the comments are very detailed. Everyone can understand it without special explanation. Can it work normally? The answer is obviously no.
Pass Jmeter simulate the following tests:

• 5000 2000 concurrent simulated users spike, will find mysql reported timeout errors while seckill_goods table amount field becomes negative, orders table also produced a record of more than 200 (will be different under different environmental specific data), which It means that it’s oversold, which doesn’t comply with the spike rules.
• Simulating a single user with 10,000 concurrent orders , more than 1 record is generated in the 060d470bed8618 table (the specific data will vary in different environments), which means that a user has bought multiple times for this event, which does not comply with the spike rules

Analysis of the code will find the problems:

• Step 2. Determine whether this product has been snapped up

  Query directly in mysql , because in the spike scenario, the concurrency will be very high, and a large number of requests are sent to the database. Obviously mysql not be supported. After all, mysql can only support thousands of concurrent requests per second.

• Step 7. Determine whether the logged-in user has been grabbed

  Under high concurrency, the last order of the same user has not been generated successfully, and it will still be judged as no if it is judged again whether it is grabbed. In this case, the code does not impose any restrictions on the deduction and ordering operations, so a single user is generated The purchase of multiple products does not meet the requirement that a user can only purchase once for this event

• Step 8, deduction of inventory operation

  Assuming that there are 1000 requests at the same time, the inventory of these 1000 requests is 200 when the product is determined in step 5 to determine whether the spike is completed. Therefore, these requests will perform step 8 to deduct the inventory, and the inventory will definitely become a negative number, that is Oversold

solution

After analysis, three issues need to be resolved:

1. Spike data needs to support high concurrent access
2. A user can only purchase once for this event, which is the purchase restriction issue
3. Reduce inventory cannot be deducted into a negative number, and the number of orders cannot exceed the set inventory number, which is an oversold problem

Redis can support high concurrency due to its high-speed processing of requests. In response to oversold, that is, inventory deductions and negative numbers, Redis can provide Lua scripts to ensure atomicity and distributed locks to solve the problem of data inconsistency under high concurrency. In response to the requirement that a user can only purchase once, Redis can solve the problem.
Therefore, you can try to use Redis solve the above problems. Specific operations:

• In order to support a large number of high concurrent inventory inspection requests, Redis needs to save spike activity data (that is, seckill_goods table data), so that the request can directly Redis and make a query. If there is inventory balance after the query is completed, Just deduct the inventory directly from Redis

• The inventory deduction operation Redis , but because the Redis is divided into two steps: read and write, that is, the data must be read and judged before the deduction is performed. Therefore, if these two operations are not done well In order to ensure the correctness of concurrent access, atomic operations need to be used to solve the problem. Redis provides a solution that uses the Lua script to contain multiple operations to achieve atomicity.
  The following is an explanation of the atomicity of Lua scripts in Redis official documents

  Atomicity of scripts
  Redis uses the same Lua interpreter to run all the commands. Also Redis guarantees that a script is executed in an atomic way: no other script or Redis command will be executed while a script is being executed. This semantic is similar to the one of MULTI / EXEC. From the point of view of all the other clients the effects of a script are either still not visible or already completed.

• Use Redis implement distributed locks to lock inventory and write order operations to ensure that a user can only purchase once.

Access Redis

First of all, the seckill_goods table is no longer used, and the new spike activity logic is changed Redis , the type is hash , the key rule is seckill_good_ + product id, now suppose a new key is seckill_good_1 , and the value is bed

{
    amount: 200,
    start_time: '2020-06-20 00:00:00',
    end_time: '2023-06-20 00:00:00',
    is_valid: 1,
    comment: '...',
  }

Secondly, create the lua script to ensure the atomicity of the deduction operation. The content of the script is as follows

if (redis.call('hexists', KEYS[1], KEYS[2]) == 1) then
  local stock = tonumber(redis.call('hget', KEYS[1], KEYS[2]));
  if (stock > 0) then
    redis.call('hincrby',  KEYS[1], KEYS[2], -1);
    return stock
  end;
  return 0
end;

Finally, complete the code, the complete code is as follows:

// 引入相关库
const moment = require('moment');
const Op = require('sequelize').Op;
const { v4: uuidv4 } = require('uuid');
// 引入数据库model文件
const seckillModel = require('../../dbs/mysql/models/seckill_goods');
const ordersModel = require('../../dbs/mysql/models/orders');
// 引入Redis实例
const redis = require('../../dbs/redis');
// 引入工具函数或工具类
const UserModule = require('../modules/user');
const { randomString, checkObjNull } = require('../../utils/tools/funcs');
// 引入秒杀key前缀
const { SECKILL_GOOD, LOCK_KEY } = require('../../utils/constants/redis-prefixs');
// 引入避免超卖lua脚本
const { stock, lock, unlock } = require('../../utils/scripts');

class Seckill {
  async doSeckill(ctx, next) {
    const body = ctx.request.body;
    const goodId = ctx.params.good_id;
    const accessToken = ctx.query.accessToken;
    const path = body.path;

    // 基本参数校验
    if (!accessToken || !path) { return ctx.throwException(20001, '参数错误！'); };
    // 判断此产品是否加入了抢购
    const key = `${SECKILL_GOOD}${goodId}`;
    const seckill = await redis.hgetall(key);
    if (!checkObjNull(seckill)) { return ctx.throwException(30002, '该产品并未有抢购活动！'); };
    // 判断是否有效
    if (!seckill.is_valid) { return ctx.throwException(30003, '该活动已结束！'); };
    // 判单是否开始、结束
    if(moment().isBefore(moment(seckill.start_time))) {
      return ctx.throwException(30004, '该抢购活动还未开始！');
    }
    if(moment().isAfter(moment(seckill.end_time))) {
      return ctx.throwException(30005, '该抢购活动已经结束！');
    }
    // 判断是否卖完
    if(seckill.amount < 1) { return ctx.throwException(30006, '该产品已经卖完了！'); };

    //获取登录用户信息(这一步只是简单模拟验证用户身份，实际开发中要有严格的登录注册校验流程)
    const userInfo = await UserModule.getUserInfo(accessToken);
    if (!userInfo) { return ctx.throwException(10002, '用户不存在！'); };

    // 判断登录用户是否已抢到
    const orderInfo = await ordersModel.findOne({
      where: {
        user_id: userInfo.id,
        good_id: goodId,
        status: { [Op.between]: ['0', '1'] },
      },
    });
    if (orderInfo) { return ctx.throwException(30007, '该用户已抢到该产品，无需再抢！'); };
    
    // 加锁，实现一个用户针对这次活动只能购买一次
    const lockKey = `${LOCK_KEY}${userInfo.id}:${goodId}`; // 锁的key有用户id和商品id组成
    const uuid = uuidv4();
    const expireTime = moment(seckill.end_time).diff(moment(), 'minutes'); // 锁存在时间为当前时间和活动结束的时间差
    const tryLock = await redis.eval(lock, 2, [lockKey, 'releaseTime', uuid, expireTime]);
    
    try {
      if (tryLock === 1) {
        // 扣库存
        const count = await redis.eval(stock, 2, [key, 'amount', '', '']);
        if (count <= 0) { return ctx.throwException(30006, '该产品已经卖完了！'); };

        // 下单
        const orderData = {
          order_no: Date.now() + randomString(4), // 这里就用当前时间戳加4位随机数作为订单号，实际开发中根据业务规划逻辑 
          good_id: goodId,
          user_id: userInfo.id,
          status: '1', // -1 已取消, 0 未付款， 1 已付款， 2已退款
          order_type: '2', // 1 常规订单 2 秒杀订单
          // seckill_id: seckill.id, // 秒杀活动id, redis中不维护秒杀活动id
          comment: '', // 备注
        };
        const order = ordersModel.create(orderData);

        if (!order) { return ctx.throwException(30008, '抢购失败!'); };
      }
    } catch (e) {
      await redis.eval(unlock, 1, [lockKey, uuid]);
      return ctx.throwException(30006, '该产品已经卖完了！');
    }

    ctx.send({
      path,
      data: '抢购成功!'
    });
  }

}

module.exports = new Seckill();

There are four main modifications to the code here:

1. Step 2. Determine whether the product has been snapped up, and check Redis
2. Step 7. Determine whether the logged-in user has been grabbed, because the snap-purchase activity id not maintained, so use user id , product id and status status
3. Step 8, deduct the inventory, use the lua script to deduct the inventory in Redis
4. Lock stock deduction and write database operations

The order operation is still Mysql database, because most of the requests were intercepted in step 5, and the remaining requests Mysql are fully capable of processing.

Tested through Jmeter again, and found that the order form is normal, and the inventory deduction is normal, indicating that the oversold problem and purchase restriction have been resolved.

other problems

1. Other technologies for spike scenarios
Based on the Redis supporting high concurrency, key-value pair database and supporting atomic operations, Redis used as a spike response solution in the case. In more complex spike scenarios, in addition to using Redis , some other technologies are needed when necessary:

   • Current limiting, using funnel algorithm, token bucket algorithm, etc. for current limiting
   • Cache, cache hot data in memory to ease the pressure of database access as much as possible
   • Peak clipping, using message queue and caching technology to transform the instantaneous high traffic into a period of stable traffic. For example, after a customer snaps up a purchase, the response is returned immediately, and then the subsequent steps are processed asynchronously through the message queue, sending text messages, writing logs, and updating consistency is low Database and so on
   • Asynchronously, suppose the merchant creates a flash campaign for fans only. If the merchant has fewer fans (assuming less than 1000), the flash campaign will be directly pushed to all fans. If the user has more fans, the program will immediately be pushed to the top 1000 users. The rest of the users use the message queue to postpone the push. (The number of 1000 needs to be determined according to specific circumstances. For example, 99% of businesses with less than 2000 fans, and only 1% of users with more than 2000 fans, then this value should be set to 2000)
   • Offloading, a single server fails to go to the cluster, and the load balance is used to process requests together to distribute the pressure

   The application of these technologies will make the entire spike system more perfect, but the core technology is still Redis . It can be said that Redis is sufficient to deal with most scenarios.

2. Redis robustness
The case uses the stand-alone version Redis , and a single node is basically not used in the production environment, because

   • Can not achieve high availability
   • Even with AOF logs and RDB snapshot solutions to ensure data is not lost, but can only be placed master on, once the machine malfunction, the service will not run, and even to take the appropriate measures are still inevitably cause data loss.

   Therefore, the master-slave mechanism and cluster mechanism of Redis

3. Redis distributed lock problem

   • Single-point distributed locks, the distributed locks mentioned in the case, are actually more accurate as single-point distributed locks, for the convenience of demonstration. However, single-point Redis distributed locks definitely cannot be used in a production environment. Reasons Similar to point 2
   • Distributed locks based on the master-slave mechanism (multi-machine) are also not enough, because redis is done asynchronously during master-slave replication. For example, after clientA acquires the lock, the master redis crashes during the process of copying data to slave redis , leading to the lock is not copied to the redis , and then from redis elect a new master redis , create a new master redis not clientA lock set, then clientB attempt to acquire the lock, and is able to successfully acquire the lock, leading to mutually exclusive failure.

   In response to the above problems, redis officially designed Redlock . The corresponding resource library Node.js node-redlock , which can be npm . At least 3 independent servers or clusters are required to use, providing a very high fault tolerance rate. In a production environment This scheme should be used first for deployment.

to sum up

The characteristics of the spike scenario can be summarized as instantaneous concurrent access, more reads and less writes, limited time and limited time. In the development, we must also consider avoiding oversold phenomena and purchase restrictions similar to scalper grabbing. In view of the above characteristics and problems, the analysis and development principles are: ：Data is written into memory instead of hard disk, asynchronous processing instead of synchronous processing, atomic execution of deduction of inventory operation, and locking of single-user purchases, and Redis is a tool that meets all the above characteristics, so Redis finally chosen to solve the problem .

The spike scenario is a relatively complex scenario in the e-commerce business. This article only introduces the core logic. The actual business may be more complicated, but it only needs to be expanded and optimized on the basis of this core.
The solution to the spike scenario is not only suitable for spikes, but also for grab red envelopes, coupons, tickets, etc. The ideas are the same.
The idea of the solution can also be applied to many scenarios, such as individual purchase restrictions, half price of the second item, and inventory control, etc. Everyone should use them flexibly.

project address

https://github.com/threerocks/redis-seckill

Reference

https://time.geekbang.org/column/article/307421
https://redis.io/topics/distlock