Vivo Global Mall: Design and Practice of Coupon System Architecture

1. Business background

Coupon is a common marketing method for e-commerce, with flexible characteristics, can be used as a carrier of promotional activities, but also an important entrance. The coupon system is an important part of the vivo mall marketing module. As early as 15 years when the vivo mall was a single application, the coupon was one of the core modules. With the development of the mall and the increase in the number of users, the coupon service has been split, and an independent coupon system has been established to provide universal coupon services. At present, the coupon system covers the four core points of coupons: creation, distribution, use, and calculation.

• "Creation" refers to the creation of coupons, including the configuration of various coupon rules and usage thresholds.
• "issue" refers to the issuance of coupons. The coupon system provides a variety of ways to issue coupons to satisfy active and passive distribution for different groups of people.
• "Use" refers to the use of coupons, including the purchase of goods in the forward direction and the refund of the coupon after the reverse refund.
• "Count" refers to the statistics of coupons, including the number of coupons issued, the number of use, the use of goods and other data summary.

In addition to the common coupon promotion gameplay, the vivo mall coupon system also uses coupons as a carrier for other activities or assets, such as the value preservation and renewal of mobile phone products, in-app purchase benefits, and cooperation with external advertisers to issue discounts Coupons and so on.

The following is the display of some scenes of vivo mall coupons:

2. System architecture and changes

Coupons were first coupled with the mall in a system. With the continuous development of the vivo mall, marketing activities have increased, and coupon use scenarios have increased. The coupon system has gradually begun to be "powerless", exposing many problems:

• The issuance of a large number of coupons has reached the bottleneck of coupon library and single table storage.
• The high coupling with the mall system directly affects the performance of the mall's entire station interface.
• The iterative update of coupons is limited by the version arrangement of the mall.
• Regarding multi-category coupons, the technical aspect does not have the ability to precipitate universal coupons.

In order to solve the above problems, the coupon system in 19 years has been system-independent and provides universal coupon services. The system architecture after independence is as follows:

Coupon system independent migration plan

How to migrate coupons from the mall system and compatible with the business parties and historical data that have been docked is also a major technical challenge. There are two options for system migration: migration with shutdown and migration without shutdown.

We use a non-stop migration plan:

• Before the migration, operations stop background operations related to coupons to avoid generating static coupon data.

Static data: The data generated in the background of the coupon, which has nothing to do with the user.

Dynamic data: coupon data related to the user, including the relationship data between coupons received by the user, coupons and orders, etc.

• Configure the current database switch to be single write, that is, the coupon data is written to the mall library (old library).
• The coupon system is online, and static data is migrated through scripts. After migration, verify the accuracy of static data migration.
• Configure the current database switch to double write, that is, online data is written into the mall library and the new coupon library at the same time. At this time, the data source provided by the service is still the mall library.
• Migrate dynamic data. After migration, verify the accuracy of dynamic data migration.
• Switch the data source, the data source provided by the service is switched to the new database. Verify that the service is correct and switch back to the mall data source when there is a problem.
• Turn off double writing, and the coupon system migration is complete.

The topological diagram of the coupon system request after migration is as follows:

Three, system design

3.1 Coupon sub-database sub-table

With the increasing volume of coupons issued, single tables have reached a bottleneck. In order to support the development of the business, comprehensive consideration, the user coupon data is divided into database and table.

Keywords: technology selection, sub-database and sub-table factor

There are mature open source solutions for sub-library and sub-table, so I won't introduce too much here. Refer to the previous project experience and adopt the self-research framework provided by the company's middleware team. The principle is to introduce the self-developed MyBatis plug-in, calculate different library table suffixes according to the customized routing strategy, and locate the corresponding library table.

The user coupon is associated with the user id, and the user id is an important field throughout the entire system, so the user id is used as the routing factor of the sub-database and sub-table. This can ensure that the same user is routed to the same database table, which not only facilitates data aggregation, but also facilitates user data query.

Assuming that there are a total of N databases and M tables, the routing strategy for database and table splitting is:

Database suffix databaseSuffix = hash(userId) / M %N

Table suffix tableSuffix = hash(userId)% M

3.2 Coupon issuance method design

In order to meet the coupon issuance requirements of various different scenarios, the coupon system provides three coupon issuance methods: unified coupon interface 16111ddafc8097, background directed coupon , coupon code exchange and issuance .

3.2.1 Unified coupon collection interface

guarantees the accuracy of coupon verification

When receiving the coupon, it is necessary to strictly verify whether the various attributes of the coupon are met: such as the recipient of the coupon, various restrictions, etc. Among them, the more critical is the verification of inventory and received quantity. Because in the case of high concurrency, it is necessary to ensure the accuracy of the quantity verification, otherwise it is easy to cause the user to over-capture.

There is a scenario: User A initiates two consecutive requests to receive coupon C, and coupon C restricts each user to receive one coupon. The first request passes the verification of the number of coupons. If the user coupons are not in the library, if there is no restriction, the second request will also pass the verification of the number of coupons. In this way, user A will successfully receive two coupons C, resulting in over-collection.

In order to solve this problem, the coupon uses a distributed lock scheme, and the implementation of distributed locks depends on Redis. Before verifying the number of coupons received by the user, try to obtain the distributed lock. After the coupon is issued successfully, the lock is released to ensure that the user will not overclaim the same coupon when the user receives the same coupon. In the above scenario, after the user successfully obtains the distributed lock for the first time, until the first request is successful to release the acquired distributed lock or timeout release, otherwise the user will fail to obtain the distributed lock for the second request, so as to ensure that A The user will only successfully receive one.

Inventory deduction

Inventory deduction is required to receive coupons. There are two common inventory deduction schemes:

Option 1: database deduction.

When deducting inventory, directly update the inventory field in the database.

The program advantage is simple and convenient, Direct Access database when you can get real-time inventory check inventory. And there is database transaction guarantee, no need to consider the problem of data loss and inconsistency.

disadvantages of are also very obvious, there are two main points:

1) Inventory is a single field in the database. When updating the inventory, all requests need to wait for the row lock. Once the amount of concurrency is large, there will be many requests blocked here, causing the request to time out, and then the system avalanche.

2) Frequent requests to the database are time-consuming and will take up a lot of database connection resources.

program two: realizes inventory deduction based on redis.

Put the inventory in the cache and use the incrby feature of redis to deduct the inventory.

advantages of the program is a breakthrough in the bottleneck of the database, with fast speed and high performance.

disadvantage of is that the system process will be more complicated, and the problem of cache loss or downtime data recovery needs to be considered, which is likely to cause inconsistent inventory data.

Considering the current and foreseeable future traffic peaks, system maintainability, and practicability of the coupon system, the coupon system adopts the improvement scheme of Scheme 1. The improvement plan is to decentralize the single inventory field into multiple inventory fields, decentralize the row lock of the database, and reduce the bottleneck of the row lock of the database when the amount of concurrency is large.

After the inventory number is updated, the inventory will be evenly distributed into M shares, and initialized and updated to the inventory record table. The user receives the coupon and randomly selects an allocated inventory field (M in total) in the inventory record table to update it. If the update is successful, the inventory deduction is successful. At the same time, the timed task will periodically synchronize the received inventory. Compared with scheme 1, this scheme breaks through the bottleneck limit of the database single row lock, and is simple to implement, without considering data loss and inconsistency.

One-click to receive multiple coupons

In the coupon collection scenario of the docked business party, there is a situation where the user can pick up multiple coupons with one click. Therefore, the unified coupon collection interface needs to support users to collect coupons with one click. In addition to receiving multiple coupons from the same coupon template, it also supports multiple coupon collections from different coupon templates. Generally speaking, receiving multiple coupons with one click refers to receiving multiple coupons with different coupon templates. In the implementation process, you need to pay attention to the following points:

1) How to ensure the performance

If you receive multiple coupons, if each coupon is separately verified, inventory deducted, and put into storage, then the bottleneck of interface performance is stuck in the number of coupons. The more the number, the performance declines linearly. So how to ensure high performance in the case of a large number of coupons? Two main measures are taken:

a. batch operation .

From the point of view of the coupon issuance process, the bottleneck lies in the warehousing of coupons. The receipt of coupons is real-time (if asynchronous, the coupons cannot be sent to the user account in real time, which will affect the user experience and the conversion rate of the coupons). The more coupons, the more IO times with the database during storage, the worse the performance . Batch warehousing can ensure that the number of IOs with the database is only once, and is not affected by the number of coupons. As mentioned above, the user coupon data is divided into database and table, and the coupon assets of the same user are stored in the same database table, so the same user can realize batch storage.

b. limits the number of coupons received at a time to .

Set the threshold and return directly after exceeding the number to ensure that the system is within the safe range.

2) Ensure that users will not overtake

If the user initiates a request in the mall, one-click to receive four coupons A/B/C/D, and the activity system issues coupon A to the user at the same time. These two coupon requests are simultaneous. Among them, coupon A restricts each user to only receive one. According to the aforementioned use of distributed locks to ensure the accuracy of verification, the keys of the distributed locks requested twice are:

User id+A\_id+B\_id+C\_id+D\_id

User id+A_id

In this case, the distributed lock requested twice does not work, because the lock key is different, and there is still the possibility of error in the quantity verification. In order to avoid the occurrence of user over-receipt during the batch collection of coupons, the acquisition of distributed locks was modified during the batch collection of coupons. In the above example, to receive four coupons of A/B/C/D with one key, you need to obtain 4 distributed locks in batches. The lock keys are:

User id+A_id

User id+B_id

User id+C_id

User id+D_id

Failure to acquire any of these locks means that the user is receiving one of the coupons at this time and needs to spin and wait (within the timeout period). After obtaining all distributed locks successfully, you can proceed to the next step.

interface idempotence

The unified coupon collection interface must ensure idempotence (idempotence: the results of one request or multiple requests initiated by the user for the same operation are consistent). In the case of network timeout or abnormal conditions, if the coupon collection result is not returned in time, the business side will retry the coupon collection. If the interface does not guarantee idempotence, it will cause overtransmission. There are many ways to realize idempotence. The coupon system uses the unique index of the database to ensure idempotence.

At the beginning, coupons did not support idempotence, and the table design did not consider idempotence.

Then needs to consider the first question: in which table to add the unique index?

There are no more than two options: existing tables or new tables.

• Using existing tables, there is no need to increase table associations. But as mentioned above, because of the sub-database and table, a large number of tables need to add unique fields, and need to be compatible with historical data, and need to ensure the uniqueness of new fields in historical data.
• Using the new table method does not need to be compatible with historical data, but the shortcomings are also obvious. The addition of a layer of table association has a great impact on performance and existing logic. Comprehensive considerations, we chose to add unique fields to the existing table, which is more conducive to ensuring performance and subsequent maintainability.

second consideration: how to compatible historical data and business side? historical data adds a unique field, which needs to be filled in with a unique value, otherwise the unique index cannot be added. We use a script to refresh the data to construct a unique value and refresh it in each row of historical data. The business party that has been docked with the coupon does not pass in a unique code. In this case, the coupon side generates a unique code as an alternative to ensure compatibility.

3.2.2 Targeted issuance

Targeted coupon issuance is used to issue coupons to specific groups of people in the background. Targeted issuance of coupons can make up for the problem of inaccurate crowd coverage and insufficient coverage when users take the initiative to receive coupons. Through targeted issuance of coupons, specific groups of people can be accurately covered and the order conversion rate can be improved. During the big promotion period, the targeted issuance of vouchers for a wide range of people can also carry the dual tasks of push activities and price reduction promotions.

Targeted coupon issuance mainly lies in the circle selection of the crowd and the design of the coupon issuance process. The overall process is as follows:

Targeted issuance is different from the user's initiative to receive coupons, and the amount of targeted issuance is usually very large (100 million). In order to support high-volume directional issuance, some optimizations have been made for directional issuance:

1) Remove transaction . The transaction logic is too heavy, and it is not necessary for directional issuance. If the issuance of coupons fails, record the failed coupons and ensure that the failure can be retried.

2) Lightweight verification . Targeted issuance limits the types of coupons, and avoids configurations that require strict verification of attributes by restricting configuration. Different from the lengthy logic of the user's active coupon verification, the verification of directional coupon issuance is very lightweight, which greatly improves the performance of coupon issuance.

3) Insert . Bulk coupon insertion reduces the number of database IOs, eliminates database bottlenecks, and increases the speed of issuing coupons. Targeted coupon issuance is for different users. User coupons are divided into libraries and tables. In order to achieve batch insertion, the library table suffixes corresponding to different users need to be calculated in the memory, and then the data is collected and then inserted in batches, up to M Times, M is the total number of library tables.

4) The core parameters can be dynamically configured . For example, the number of coupons issued in a single time, the number of library reads in a single time, the number of users contained in the message body sent to the message center, etc., can control the peak speed and average speed of directional coupon issuance.

3.2.3 Coupon code exchange

The way of issuing off-site marketing coupons is different from other coupons, and they are exchanged through coupon codes. The coupon code is exported from the background and distributed to the user through SMS or activities. The user obtains the corresponding coupon after redeeming the coupon code. There are certain rules for the composition of coupon codes, and security must be ensured on the basis of the rules. This security is mainly the accuracy of the coupon code verification, preventing the redemption of redeemed coupon codes and malicious redemption of invalid coupon codes.

3.3 Refined marketing capability design

Through the way of label combination configuration, coupons provide the ability of refined marketing to realize the diverse aspects of coupons. Tags can be divided into quasi-real-time and real-time. It is worth noting that the processing of some real-time tags requires prerequisites, such as regional attributes that require user authorization.

Accurate reach of coupons:

3.4 The relationship between coupons and commodities

The use of coupons needs to be associated with products, and can be associated with all products or part of them. In order to flexibly meet the configuration of the coupon-related commodities in the operation, the coupon system has two association methods:

a. Blacklist.

Available products = all products-blacklisted products.

The blacklist is applicable to the situation where the coupon can be used with a wide range of products, and all the products excluded from the blacklist are the coupons' available range.

b. Whitelist.

Available products = whitelisted products.

The whitelist is suitable for the situation where the range of available commodities of the coupon is relatively small, and the available commodities of the coupon are directly allocated.

In addition, there is also the configuration of the super blacklist. The blacklist and whitelist are only valid for a single coupon, and the super blacklist is valid for all coupons. The current coupon system provides product-level associations. Subsequent coupons will support the association of product classification dimensions. Classification dimensions + commodity dimensions can more flexibly associate coupons and products.

3.5 High performance guarantee

There are many coupon docking systems, and there are high-traffic scenarios. The external interface provided by coupons needs to ensure high performance and high stability.

Multi-level cache

In order to improve the query speed, reduce the pressure on the database, and at the same time to cope with scenarios where instantaneous high traffic brings hot keys (such as switching traffic to the specific product business page at the end of the live broadcast of the press conference, and the hot event product business page, it will bring to the coupon system Instantaneous high traffic), the coupon uses a multi-level cache.

Database read and write separation

In addition to the aforementioned sub-library and sub-table, the coupons also perform read-write separation operations on this basis. The main database is responsible for executing the data update request, and then synchronize the data changes to all the slave libraries in real time, and use the slave libraries to share the query request and solve the problem of database writing affecting the query. There is a delay in master-slave synchronization. Normally, the delay does not exceed 1ms. There is a time-consuming process for coupon collection or status change, and the master-slave delay is not perceptible to users.

relies on external interface isolation and fuse

The coupon internally relies on a third-party system. In order to prevent the unavailability of the relying party's service, the chain effect will eventually lead to an avalanche of the coupon service, the coupon is isolated and fused to rely on external interfaces.

User dimension coupon field redundancy

Querying user-related coupon data is one of the most frequent query operations for coupons. User coupon data is divided into database and table. When querying, it is impossible to associate the coupon rule table for query. In order to reduce the number of IOs, the user coupon table The fields of some coupon rules are redundant. There are many fields in the coupon rule table, and there cannot be many redundant fields. A balance should be made between performance and the number of fields.

Fourth, summary and outlook

Finally, a summary of the coupon system:

• Non-stop migration, smooth transition. It has been operating stably for 2 years since its independence, and its performance is sufficient to support the rapid development of vivo mall in the next 3-5 years.
• The system is decoupled, and the iteration efficiency is greatly improved.
• For business problems, the principle is to choose a suitable and practical solution.
• Possess complete coupon business capabilities.

Outlook: At present, the coupon system mainly serves the vivo mall. In the future, we hope to open up the coupon capabilities to provide a universal and integrated coupon platform for other internal business parties.

Author: vivo Internet Development Team-Yan Chao