In the Rain of Red Envelopes: The Encounter of Redis and Lua

In 2018, Wang Sicong's summit conference , the million-dollar hero of watermelon video, and then to Inke's Cheeseman , the live answering questions were popular all over the Internet.

An e-commerce company I served also joined this craze, and the technical team developed a live answering function. After answering the question, the red envelopes will fall in the form of red envelope rain . The user clicks on the red envelopes that fall on the screen. If the red envelopes are grabbed, the red envelopes will enter the user's account in the form of cash.

Red envelope rain is a typical high-concurrency scenario. There are massive requests to access the server in a short period of time. In order to make the system run smoothly, the technical team adopted a design scheme based on Redis + Lua script to grab red envelopes.

1 Overall process

Let's analyze the overall process of grabbing red packets:

1. The operating system configures the total amount of red envelopes and the number of red envelopes, calculates the amount of each red envelope in advance and stores it in Redis;
2. Grab the red envelope rain interface, the user clicks the red envelope falling on the screen to initiate a red envelope grab request;
3. After the TCP gateway receives the red packet grab request, it calls the answering system to grab the red packet dubbo service. The red packet grab service essentially executes the Lua script and returns the result to the front end through the TCP gateway;
4. If the user grabs the red envelope, the asynchronous task will obtain the red envelope information from Redis, call the balance system, and return the amount to the user account.

2 Red Packet Redis Design

The rules for grabbing red packets are as follows:

• For the same event, the user can only grab the red envelope once;
• The number of red envelopes is limited, and one red envelope can only be grabbed by one user.

As shown below, we design three data types:

1. Operation of pre-allocated red envelopes list;

Queue element json data format:

 {
    //红包编号
    redPacketId : '365628617880842241' 
    //红包金额
    amount : '12.21'          
}

2. User red envelope collection record list;

Queue element json data format:

 {
    //红包编号
    redPacketId : '365628617880842241'
    //红包金额
    amount : '12.21',
    //用户编号
    userId : '265628617882842248'
}

3. User red envelope anti-heavy Hash table;

The operation process of grabbing the red envelope Redis:

1. Use the hexist command to determine whether the user has received a red envelope in the anti-replication Hash table of the red envelope receipt record. If the user has not received a red envelope, the process continues;
2. A red packet data is generated from the operation pre-allocated red packet list rpop;
3. Operate the anti-replication Hash table of the red envelope collection record, and call the HSET command to store the user's collection record;
4. Enter the red packet receiving information lpush into the user red packet receiving record list.

In the process of grabbing red envelopes, you need to focus on the following points:

• Execute multiple commands, whether atomicity can be guaranteed, and whether a command can be rolled back if it fails to execute;
• In the execution process, in high concurrency scenarios, whether isolation can be maintained;
• Subsequent steps depend on the results of previous steps.

Redis supports two modes: transaction mode and Lua script . Next, we will expand them one by one.

3 Transaction Principles

A Redis transaction consists of the following commands:

serial number	Command and Description
1	MULTI marks the beginning of a transaction block.
2	EXEC executes commands within all transaction blocks.
3	DISCARD cancels the transaction, giving up execution of all commands within the transaction block.
4	WATCH key [key ...] Watches one (or more) keys, and if this (or these) keys are changed by other commands before the transaction is executed, then the transaction will be interrupted.
5	UNWATCH cancels the monitoring of all keys by the WATCH command.

A transaction consists of three phases:

1. The transaction is opened, using MULTI, this command marks that the client executing the command switches from the non-transactional state to the transactional state;
2. The command is queued, and after MULTI opens the transaction, the client's command will not be executed immediately, but will be put into a transaction queue;
3. Execute the transaction or discard. If an EXEC command is received, the command in the transaction queue will be executed, and if it is DISCARD, the transaction will be discarded.

An example of a transaction is shown below.

 redis> MULTI 
OK
redis> SET msg "hello world"
QUEUED
redis> GET msg
QUEUED
redis> EXEC
1) OK
1) hello world

Have a question here? Can the Redis key be modified when a transaction is started?

The Redis key can still be modified before the transaction executes the EXEC command .

Before the transaction is started, we can monitor the Redis key with the watch command. Before the transaction is executed, we modify the key value, the transaction fails, and returns nil .

Through the above example, the watch command can achieve an effect similar to optimistic locking .

4 ACID for transactions

4.1 Atomicity

Atomicity means that all operations in a transaction, either all completed or all incomplete, will not end in a certain link in the middle. If an error occurs during the execution of a transaction, it will be rolled back to the state before the transaction started, as if the transaction had never been executed.

First example:

Before executing the EXEC command, the operation command sent by the client is incorrect, such as: syntax error or using a non-existing command.

 redis> MULTI
OK
redis> SET msg "other msg"
QUEUED
redis> wrongcommand  ### 故意写错误的命令
(error) ERR unknown command 'wrongcommand' 
redis> EXEC
(error) EXECABORT Transaction discarded because of previous errors.
redis> GET msg
"hello world"

In this example, we used a non-existent command, which caused the enqueue to fail and the entire transaction to fail.

Second example:

When a transaction operation is queued, the data types of the command and the operation do not match, and the queue is normal, but the execution of the EXEC command is abnormal.

 redis> MULTI  
OK
redis> SET msg "other msg"
QUEUED
redis> SET mystring "I am a string"
QUEUED
redis> HMSET mystring name  "test"
QUEUED
redis> SET msg "after"
QUEUED
redis> EXEC
1) OK
2) OK
3) (error) WRONGTYPE Operation against a key holding the wrong kind of value
4) OK
redis> GET msg
"after"

In this example, when Redis executes the EXEC command, if an error occurs, Redis will not terminate the execution of other commands, and the transaction will not be rolled back due to a command failure.

In summary, my understanding of the atomicity of Redis transactions is as follows:

1. When an error occurs when the command is enqueued, the transaction execution will be abandoned to ensure atomicity;
2. When the command enters the queue, it is normal. After executing the EXEC command, an error is reported, and atomicity is not guaranteed;

That is: Redis transactions have certain atomicity only under certain conditions .

4.2 Isolation

The isolation of the database refers to the ability of the database to allow multiple concurrent transactions to read, write and modify its data at the same time. Isolation can prevent data inconsistency due to cross execution when multiple transactions are executed concurrently.

Transaction isolation is divided into different levels, namely:

• read uncommitted
• read committed
• repeatable read
• serializable

First of all, it needs to be clear: Redis does not have the concept of transaction isolation level. Here we discuss the isolation of Redis: whether transactions can not interfere with each other in a concurrent scenario .

We can divide transaction execution into two stages before the execution of the EXEC command and after the execution of the EXEC command, and discuss them separately.

1. Before the EXEC command is executed

In the section on transaction principle, we found that the Redis key can still be modified before the transaction is executed. At this point, the WATCH mechanism can be used to achieve the effect of optimistic locking.

2. After the EXEC command is executed

Because Redis executes operation commands in a single thread, after the EXEC command is executed, Redis will ensure that all commands in the command queue are executed. This ensures transaction isolation.

4.3 Persistence

The durability of the database means that after the transaction is completed, the modification of the data is permanent, even if the system fails, it will not be lost.

Whether the data of Redis is persistent depends on the persistent configuration mode of Redis.

1. Without RDB or AOF configured, the durability of transactions cannot be guaranteed;
2. Using the RDB mode, after a transaction is executed, before the next RDB snapshot is executed, if the instance is down, the durability of the transaction cannot be guaranteed either;
3. The AOF mode is used; the three configuration options no and everysec of the AOF mode will have data loss. always can guarantee the durability of the transaction, but because the performance is too poor, it is generally not recommended in the production environment.

In summary, the durability of redis transactions cannot be guaranteed .

4.4 Consistency

The concept of consistency has always been confusing, and in the sources I've searched, there are two different definitions.

1. Wikipedia

Let's first look at the definition of consistency on Wikipedia:

Consistency ensures that a transaction can only bring the database from one valid state to another, maintaining database invariants: any data written to the database must be valid according to all defined rules, including constraints, cascades, triggers, and any combination thereof. This prevents database corruption by an illegal transaction, but does not guarantee that a transaction is correct. Referential integrity guarantees the primary key – foreign key relationship.

In this text, the core of consistency is " constraint ", " any data written to the database must be valid according to all defined rules ".

How to understand constraints? Here is a quote from Zhihu Question How to understand the internal consistency and external consistency of the database, a passage answered by Han Fusheng, an expert in OceanBase R&D of Ant Financial:

"Constraints" are told to the database by the user of the database, and the user requires that the data must conform to this or that constraint. When the data is modified, the database will check whether the data still meets the constraints. If the constraints are no longer met, the modification operation will not occur.

The two most common types of constraints in relational databases are "uniqueness constraints" and "integrity constraints". The primary and unique keys defined in the table ensure that the specified data item will never be repeated, and the referential integrity defined between the tables It also ensures the consistency of the same attribute in different tables.

"Consistency in ACID" is so easy to use that it has melted into the blood of most users. Users will consciously add required constraints when designing tables, and the database will strictly enforce this constraint condition.

Therefore , the consistency of the transaction is related to the predefined constraints, and ensuring the constraints guarantees the consistency .

Let's take a closer look at this sentence: This prevents database corruption by an illegal transaction, but does not guarantee that a transaction is correct .

It may be a little vague for everyone to write here, let's take the case of a classic transfer .

We start a transaction. The initial balances on Zhang San and Li Si's accounts are both 1,000 yuan, and the balance field has no constraints. Zhang San transferred 1,200 yuan to Li Si. Zhang San's balance is updated to -200, and Li Si's balance is updated to 2200.

From the application level, this transaction is obviously illegal, because in real scenarios, the user balance cannot be less than 0, but it completely follows the constraints of the database, so from the database level, this transaction still guarantees consistency.

The transaction consistency of Redis means that the Redis transaction conforms to the constraints of the database during execution and does not contain illegal or invalid error data.

We discuss it in three abnormal scenarios:

1. Before executing the EXEC command, the operation command sent by the client is incorrect, the transaction is terminated, and the data remains consistent;
2. After the EXEC command is executed, the data types of the command and the operation do not match, and the wrong command will report an error, but the transaction will not be terminated due to the wrong command, but will continue to execute. The correct command is executed normally, and the wrong command reports an error. From this point of view, the data can also be kept consistent;

3. During the execution of the transaction, the Redis service is down. The persistence mode of the service configuration needs to be considered here.

   • Non-persistent memory mode: After the service is restarted, the database does not maintain data, so the data is consistent;
   • RDB / AOF mode: After the service is restarted, Redis restores data through RDB / AOF files, and the database is restored to a consistent state.

To sum up, under the meaning that the core of consistency is constraint, Redis transactions can guarantee consistency .

2. Designing Data-Intensive Applications

This book is a godsend for getting started with distributed systems. There is an explanation of ACID in the transaction chapter:

Atomicity, isolation, and durability are properties of the database, whereas consistency (in the ACID sense) is a property of the application. The application may rely on the database's atomicity and isolation properties in order to achieve consistency, but it's not up to the database alone. Thus, the letter C doesn't really belong in ACID.

Atomicity, isolation, and durability are properties of databases, while consistency (in the ACID sense) is properties of applications. Applications may rely on the atomicity and isolation properties of the database to achieve consistency, but this is not only dependent on the database. Therefore, the letter C is not part of ACID.

Many times, the consistency we have been struggling with actually refers to consistency in the real world, and consistency in the real world is the ultimate goal of affairs.

In order to achieve real-world consistency, the following points need to be met:

1. Guarantee atomicity, durability and isolation. If these characteristics cannot be guaranteed, then the consistency of the transaction cannot be guaranteed;
2. The constraints of the database itself, such as the length of the string cannot exceed the limit of the column or the uniqueness constraint;
3. The business level also needs to be guaranteed.

4.5 Summary

We usually call Redis an in-memory database, which is different from traditional relational databases. In order to provide higher performance and faster writing speed, some balances have been made at the design and implementation level, and it cannot fully support ACID of transactions.

Redis transactions have the following characteristics:

• guarantee isolation;
• Durability is not guaranteed;
• It has a certain atomicity, but does not support rollback;
• The concept of consistency is divergent. It is assumed that Redis transactions can guarantee consistency under the meaning that the core of consistency is constraint.

In addition, in the scenario of grabbing red envelopes, because each step needs to rely on the result returned by the previous step, it is necessary to implement optimistic locking through watch. From an engineering point of view, Redis transactions are not suitable for this business scenario.

5 Lua scripts

5.1 Introduction

"Lua" means "moon" in Portuguese and was developed in 1993 by the Pontifical Catholic University in Brazil.

The language is designed to be embedded in applications to provide flexible extension and customization capabilities for applications.

Lua scripts can be easily called by C/C++ code, and can also call C/C++ functions in turn, which makes Lua widely used in applications. Not only as an extension script, but also as a common configuration file, instead of XML, Ini and other file formats, and easier to understand and maintain.

Lua is written in standard C, the code is concise and beautiful, and can be compiled and run on almost all operating systems and platforms.

A complete Lua interpreter is only 200k. Among all the current scripting engines, Lua is the fastest. All this determines that Lua is the best choice for embedded scripts.

Lua scripts shine in the game field. The well-known "Westward Journey II" and "World of Warcraft" all use Lua scripts a lot.

The api gateways that Java back-end engineers have contacted, such as Openresty and Kong , can see Lua scripts.

Starting from Redis 2.6.0, the built-in Lua interpreter in Redis can run Lua scripts in Redis.

Benefits of using Lua scripts:

• Reduce network overhead. Send multiple requests in the form of scripts at one time to reduce network latency.
• Atomic operation. Redis will execute the entire script as a whole, without being intervened by other commands.
• reuse. The script sent by the client is permanently stored in Redis, and other clients can reuse the script without using code to complete the same logic.

Common commands for Redis Lua scripts:

serial number	Command and Description
1	EVAL script numkeys key [key ...] arg [arg ...] Execute a Lua script.
2	EVALSHA sha1 numkeys key [key ...] arg [arg ...] Execute a Lua script.
3	SCRIPT EXISTS script [script ...] Checks whether the specified script is already stored in the cache.
4	SCRIPT FLUSH Removes all scripts from the script cache.
5	SCRIPT KILL kills the currently running Lua script.
6	SCRIPT LOAD script adds the script script to the script cache, but does not execute the script immediately.

5.2 EVAL command

Command format:

 EVAL script numkeys key [key ...] arg [arg ...]

illustrate:

• script is the first parameter, which is a Lua 5.1 script;
• The second parameter numkeys specifies several keys for subsequent parameters;
• key [key ...] , is the key to be operated, multiple keys can be specified, and obtained in the Lua script by KEYS[1] , KEYS[2] ;
• arg [arg ...] , parameter, obtained by ARGV[1] , ARGV[2] in Lua script.

Simple example:

 redis> eval "return ARGV[1]" 0 100 
"100"
redis> eval "return {ARGV[1],ARGV[2]}" 0 100 101
1) "100"
2) "101"
redis> eval "return {KEYS[1],KEYS[2],ARGV[1]}" 2 key1 key2 first second
1) "key1"
2) "key2"
3) "first"
4) "second"

The following demonstrates how Lua calls the Redis command, and executes the Redis command through redis.call() .

 redis> set mystring 'hello world'
OK
redis> get mystring
"hello world"
redis> EVAL "return redis.call('GET',KEYS[1])" 1 mystring
"hello world"
redis> EVAL "return redis.call('GET','mystring')" 0
"hello world"

5.3 EVALSHA command

The Lua script needs to be transmitted for each request using the EVAL command. If the Lua script is too long, it will not only consume network bandwidth, but also have a certain impact on the performance of Redis.

The idea is to cache the Lua script first, and return the sha1 summary of the client Lua script. The client stores the sha1 digest of the script and executes the EVALSHA command for each request.

The basic syntax of the EVALSHA command is as follows:

 redis> EVALSHA sha1 numkeys key [key ...] arg [arg ...]

Examples are as follows:

 redis> SCRIPT LOAD "return 'hello world'"
"5332031c6b470dc5a0dd9b4bf2030dea6d65de91"
redis> EVALSHA 5332031c6b470dc5a0dd9b4bf2030dea6d65de91 0
"hello world"

5.4 Transactions vs Lua Scripts

By definition, a script in Redis is itself a transaction, so anything that can be done in a transaction can also be done in a script. And in general, using scripts is easier and faster.

Because the script function was only introduced in Redis 2.6, and the transaction function existed earlier, Redis has two methods for handling transactions at the same time.

However, we do not intend to remove transaction functionality anytime soon, because transactions provide a way to avoid race conditions even without scripting, and transactions themselves are not complicated to implement.

-- https://redis.io/

Lua script is another form of transaction, it has certain atomicity, but in the case of script error, the transaction will not be rolled back. Lua scripts can guarantee isolation, and can perfectly support subsequent steps relying on the results of previous steps .

To sum up, Lua script is the best solution for grabbing red envelopes.

But when writing Lua scripts, pay attention to the following two points:

1. In order to avoid Redis blocking, Lua script business logic should not be too complicated and time-consuming;
2. Check and test Lua scripts carefully, because the execution of Lua scripts is atomic and does not support rollback.

6 Preparation for actual combat

I choose Redisson version 3.12.0 as the Redis client, and make a thin layer of encapsulation based on the Redisson source code.

Create a PlatformScriptCommand class to execute Lua scripts.

 // 加载 Lua 脚本 
String scriptLoad(String luaScript);
// 执行 Lua 脚本
Object eval(String shardingkey, 
            String luaScript, 
            ReturnType returnType,
            List<Object> keys, 
            Object... values);
// 通过 sha1 摘要执行Lua脚本
Object evalSha(String shardingkey, 
               String shaDigest,
               List<Object> keys, 
               Object... values);

Why do we need to add a shardingkey parameter here?

Because in Redis cluster mode, we need to locate which node to execute the Lua script.

 public int calcSlot(String key) {
    if (key == null) {
        return 0;
    }
    int start = key.indexOf('{');
    if (start != -1) {
        int end = key.indexOf('}');
        key = key.substring(start+1, end);
    }
    int result = CRC16.crc16(key.getBytes()) % MAX_SLOT;
    log.debug("slot {} for {}", result, key);
    return result;
}

7 Grab the red envelope script

After the client executes the Lua script, it returns a json string.

• The user successfully grabbed the red envelope

 {
    "code":"0",
    //红包金额   
    "amount":"7.1",
    //红包编号
    "redPacketId":"162339217730846210"
}

• User has received

 {
    "code":"1"
}

• The user failed to grab the red envelope

 {
    "code":"-1"
}

Redis Lua has a built-in cjson function for json encoding and decoding.

 -- KEY[1]: 用户防重领取记录
local userHashKey = KEYS[1];
-- KEY[2]: 运营预分配红包列表
local redPacketOperatingKey = KEYS[2];
-- KEY[3]: 用户红包领取记录 
local userAmountKey = KEYS[3];
-- KEY[4]: 用户编号
local userId = KEYS[4];
local result = {};
-- 判断用户是否领取过 
if redis.call('hexists', userHashKey, userId) == 1 then
  result['code'] = '1'; 
  return cjson.encode(result);
else
   -- 从预分配红包中获取红包数据
   local redPacket = redis.call('rpop', redPacketOperatingKey);
   if redPacket
   then
      local data = cjson.decode(redPacket);
      -- 加入用户ID信息
      data['userId'] = userId; 
     -- 把用户编号放到去重的哈希，value设置为红包编号
      redis.call('hset', userHashKey, userId, data['redPacketId']);
     --  用户和红包放到已消费队列里
      redis.call('lpush', userAmountKey, cjson.encode(data));
     -- 组装成功返回值
      result['redPacketId'] = data['redPacketId'];
      result['code'] = '0';
      result['amount'] = data['amount'];
      return cjson.encode(result);
   else
      -- 抢红包失败
      result['code'] = '-1';
      return cjson.encode(result);
   end 
end

In the process of script writing, it is inevitable that there will be omissions, how to debug?

I personally recommend a combination of the two methods.

1. Write junit test cases;
2. Starting from Redis 3.2, Lua debugger (abbreviation LDB ) is built-in, you can use Lua debugger to debug Lua scripts.

8 Asynchronous tasks

Two classes are encapsulated on the basis of Redisson to simplify the use cost of developers.

1. RedisMessageConsumer : Consumer class , configure the name of the listening queue, and the corresponding consumption listener

 String groupName = "userGroup";
String queueName = "userAmountQueue";
RedisMessageQueueBuilder buidler =
        redisClient.getRedisMessageQueueBuilder();
RedisMessageConsumer consumer =
        new RedisMessageConsumer(groupName, buidler);
consumer.subscribe(queueName, userAmountMessageListener);
consumer.start();

2. RedisMessageListener : Consumption listener , write business consumption code

 public class UserAmountMessageListener implements RedisMessageListener {
  @Override
  public RedisConsumeAction onMessage(RedisMessage redisMessage) {
   try {
    String message = (String) redisMessage.getData();
    // TODO 调用用户余额系统
    // 返回消费成功
    return RedisConsumeAction.CommitMessage;
   }catch (Exception e) {
    logger.error("userAmountService invoke error:", e);
    // 消费失败，执行重试操作
    return RedisConsumeAction.ReconsumeLater;
  }
 }
}

9 to the end

"On paper, it's superficial, and I don't know what to do ."

In the process of learning Redis Lua, I have queried a lot of information, practiced one example and one example, and gained a lot.

To be honest, before writing this article, I had a lot of assumptions about Redis Lua. For example, I was surprised that Redis transactions could not be rolled back.

Therefore, when faced with knowledge points that you are not familiar with, don't jump to conclusions, and learn with a humble attitude, which is the attitude an engineer needs.

At the same time, no technology is perfect, there is a balance of one kind or another between design and coding, and this is the real world.

---

If my article is helpful to you, please like , watch, and forward it. Your support will inspire me to output higher-quality articles, thank you very much!