618 Technical Special (2) Why is it more and more easy to grab a spike where millions of people place orders at the same time?

Summary: people place orders at the same time. Why do you get more and more probabilities? How do e-commerce companies grasp the sales data of hundreds of millions of dollars during the big promotion period?

This article is shared from the HUAWEI CLOUD community " 618 Technical Special (2) seckill where millions of people place orders at the same time, why is it more and more easy to grab them? ", original author: technical torchbearer.

Add your favorite product to the shopping cart, click on settlement, payment, product page sales +1, inventory -1, if only dozens of people place the order, the system can easily handle it, but when the number becomes millions or tens of millions, it is It's another situation.

E-commerce companies face traffic access pressure during the big promotion: the surge of website users has overloaded the single server, which will cause the website to or fail, which will seriously affect the user experience. 160d2aac1be091 elastic load balancing service can easily solve this problem.

Elastic load balancing: where to move under pressure

Elastic Load Balance (ELB) is a traffic distribution control service that distributes access traffic to multiple back-end servers according to a forwarding strategy. HUAWEI CLOUD’s ELB can expand the external service capabilities of the application system through traffic distribution and pass Eliminate single points of failure to improve the availability of application systems.

In the face of different e-commerce business needs, ELB can handle it flexibly. For example, for a business with a large amount of business visits, you can set up corresponding forwarding rules through ELB to evenly divide the traffic into multiple back-end cloud servers for processing; for businesses with tidal effects, you can add them to ELB at any time And the removal of back-end cloud servers, such as adding servers before the first wave of the big promotion, removing them in the mid-term, and adding them again at the end.

This can ensure that during the big promotion period, the product page will not be blocked due to excessive visits, and it can also control the cost of expenditure. But how do the operators know when to add/remove servers, and when to set forwarding rules for ELB, they have to have a basic prediction on the rhythm of the entire promotion and their own product data.

Therefore, during the big promotion period, in addition to the pressure of access traffic, the real-time data analysis and management capabilities are even more troublesome.

Data warehouse: if there is no stock, how many orders are sold, the system door is clear

Under normal circumstances, e-commerce data is mainly divided into two parts:

• The first category is transaction-oriented orders, commodities, activities, shipments and other data;
• The second category is the real-time log of platform operation and the behavior data generated by users' activities on the platform.

If companies cannot cope with the business requirements of large-scale online transactions and "real-time analysis", they will not be able to take the initiative in such a big promotion such as "618".

For example, the original system of a sportswear e-commerce company uses traditional solutions, and transactions and BI are independent of each other. The trading platform is built with distributed middleware + stand-alone database. Since this solution does not have strong data consistency capabilities, the data in the system may be incomplete and inaccurate at the same time. For example, 100 orders have been sold one minute ago, but the inventory has not changed, which will finally bring extreme value to sales. Great difficulty. In order to ensure the final consistency of the data, the transaction system data needs to be synchronized to the BI system after a few hours through the ETL tool. Real-time analysis cannot be achieved, and the sales and operation supervisors cannot grasp the business situation in real time.

To solve this problem, they later adopted the Huawei Cloud Hybrid Load Data Warehouse DWS. DWS adopts the design concept of "one database for two purposes". A set of data warehouse clusters can support ultra-high concurrency and low latency business transaction requests, and at the same time support complex massive data analysis and BI applications, reducing development and operation and maintenance costs . Compared with the original system, the timeliness of the BI system is greatly improved, and the data analysis performance is improved by 3 times.

While achieving real-time data consistency, DWS also ensures the accuracy of single data and the "zero" delay of operating reports.

The small-scale DWS can support the write capability of 10,000-level TPS, and after horizontal expansion, it can reach millions to tens of millions of TPS, supporting the e-commerce enterprise to smoothly pass the "Double Eleven" and "Double 12". And because of its native ACID features of distributed transactions and strong data consistency guarantee, it can guarantee the accuracy and completeness of the transaction system data at any time, and ensure that the single data is correct. Coupled with the high-availability architecture design of DWS, it provides automatic data backup function, and the reliability reaches 11 decimal points to ensure that business data is not lost.

Data visualization: Whether there is a record-breaking sales, it depends on it

When we integrate the data of different system modules into the data warehouse, let it further clean, integrate, and process the rules to achieve real-time, complete and accurate data, we need to push it to the product and operation staff through the data visualization platform for visual analysis.

Huawei Mall previously relocated the e-commerce big data application from the TIDB+Spark cluster to the data warehouse platform based on Huawei Cloud with the DWS database as the core.

In terms of real-time data integration, Huawei Cloud provides CS stream computing services based on Flink and Spark two stream computing engines. With the activation of the Flink community and the lowering of FlinkSQL's great threshold for application developers, stream computing can be realized in the form of SQL.

HUAWEI CLOUD encapsulates general message middleware into source operators corresponding to data input. The logic layer is represented by SQL, and various data storage platforms are encapsulated as sink operators. If you use the native Flink API to develop and publish a short input and output task, at least 2 hours can be completed, but now the one-stop streaming computing platform can be completed in 10 minutes. and after grammatical verification, most of the problems can be circumvented in this link. After publishing, you can view the input and output data information through the data visualization interface for verification and monitoring.

The BI data visualization module chooses the Huawei Cloud DLV service, which focuses on large-screen scenarios. A big promotion event in Huawei Mall needed to temporarily add a command big screen to view the data. Two developers added a 2-hour shift, one for the front desk and the other for data processing logic. Version 0.1 was quickly completed and released, and then iterative optimization was continued. effect.

Picture: A company's operating data dashboard

In addition to the core data warehouse equipment, the DAYU data development kit platform can greatly improve the release efficiency. DAYU takes the data scheduling platform as the core as an extension, and integrates data monitoring, metadata management, and data service release. These services provide the big data platform of Huawei Mall. There are gradual use.

Briefly introduce the most commonly used DLF (scheduling platform) among developers. In terms of a data API release, the developer first opens a job on the DLF and drags in three tasks: CDM (integrate TIDB tasks into DWS), 2. One DWS SQL (one for DWR layer rule processing, one for data DWD result presentation), and then publish the result table data through the API of Dayu’s DLG service, which can be called by other fields.

After e-commerce companies go to the cloud, they have solved the problem of infrastructure and can focus more on business logic development, without having to consider service load balancing, disaster tolerance, etc., and also reduce the burden of operation and maintenance and labor costs. For consumers, the rush-buying experience during the big promotion period is better, and they can grab their favorite products by placing orders during peak traffic periods.

At last

In the battle for consumers for e-commerce, technological advancement is of utmost importance. From the generation of consumer demand to the in-depth purchase behavior, from the supply chain to the analysis and management of platform transaction data, from the low latency of live streaming to the security of business data, the iteration of cloud computing, AI, big data and other technologies, one step Changing the online shopping model has also defined a new business model.

In the past, AR/VR technology provided new possibilities for content presentation and consumer interaction. With the maturity of 5G technology and the high investment and trial and error in the early stage, VR shopping that simulates real scenes may not be enough. Going out and shopping while shopping has become a normal shopping mode.

At the other end of the consumer-the smart logistics side, IoT, edge computing, machine vision, and unmanned driving. These technologies are already changing the traditional logistics warehousing and transportation system, from automated three-dimensional warehousing, automatic transportation, automatic sorting to robots Work, the standardized and unified automatic operation improves the operational efficiency.

618 is a shopping festival for consumers. It is also a big test for e-commerce companies. The new high performance tests their technical strength. From another perspective, while stimulating domestic demand, 618 also tightened the rope behind technological innovation and industrial upgrading.

618 Technical Special (1) Unknowingly, it exceeded the budget by 3 times, why can't you stop buying and buying? If you want to understand why our wallets are always emptied whenever there is a big sale? Behind this is whether everyone's self-control is not enough, or the e-commerce platform is too good at understanding people's hearts, we might as well take a look at it from the technical dimension.

Click to follow and learn about Huawei Cloud's fresh technology for the first time~