Abstract: When e-commerce platforms demand more and more AI algorithm models, the unified construction of feature data platforms is a headache for many development teams. Because only through unified feature data storage, can the original "data island" be changed and the dilemma of repeated production wheels can be solved.
This article is shared from Huawei Cloud Community " Huawei Cloud Enterprise Redis: Helping VMALL Build an Advanced Feature Platform ", author: GaussDB database.
1 Customer introduction
Huawei Mall (VMALL) is the official e-commerce platform of Huawei’s self-operated and selected good things. Based on the concept of "smart life, selected good things", it provides consumers with the most complete range of Huawei brand products and Hongmeng ecological products, covering In order to meet the daily needs of office, travel, home, sports, entertainment, etc., we are committed to bringing smart life in all scenarios to more consumers.
The cloud database GaussDB (for Redis), as an enterprise Redis under Huawei Cloud, is committed to providing customers with stable, reliable, ultra-high concurrency, and extremely fast and elastic KV storage services. GaussDB (for Redis) has played a key role in the construction of the VMALL feature engineering platform.
2 Business pain points
VMALL uses a large number of AI and big data technologies to support the efficient development of services such as smart recommendation, precision marketing, smart search, and product selection.
With the rapid development of business, the system has an increasing demand for AI algorithm models. In the current AI development process, both the "model training" and "model deployment" phases are already supported by mature platforms, but the "feature data preparation" phase lacks a common platform, resulting in "feature data for online reasoning and offline training" "Inconsistency", "Independent development of each algorithm model, and repeated wheel creation of feature production", "High investment in feature engineering (accounting for 60%-70% of algorithm development time)" 3 key issues, which seriously affect R&D efficiency and hinder business develop.
To solve this problem, the VMALL big data team started to build a unified feature platform.
The core component of the feature platform is the feature storage database. Only through unified feature data storage can the original "data island" dilemma be changed, and the three major problems of "inconsistency", "difficult to share" and "low efficiency" can be completely solved.
However, it is precisely because the feature database needs to be responsible for opening up multiple online/offline scenarios, connecting batch/streaming multiple data sources, and meeting the diverse consumption needs of training/reasoning, which puts forward very high requirements for the selection of the feature database : Need to find a data storage service. can not only provide low-cost mass data storage and facilitate expansion, but also ensure the absolute reliability of data and the high availability of services; it must not only meet the low-latency online reasoning, but also meet High-throughput offline training; not only can provide a simple KV interface for easy downstream consumption, but also compatible with mainstream batch/stream processing engines (Spark/Flink, etc.) for fast upstream access.
After in-depth research, the VMALL big data team finally chose GaussDB (for Redis) as the feature database. Let's take a closer look at how GaussDB (for Redis) meets the above-mentioned demanding requirements.
3 Solution
The main process of the feature platform using GaussDB (for Redis)
1) Feature production (extraction, processing, storage)
- Offline features (static features):
Schedule Spark jobs regularly, extract data from various data warehouses and data lakes, perform feature engineering processing, and store them in GaussDB (for Redis).
- Real-time characteristics (dynamic characteristics):
Flink consumes data in Kafka, or streaming storage, and continuously updates to GaussDB (for Redis).
2) Feature consumption
- Online reasoning:
The model has been deployed to production and started to undertake business. It requires low-latency, high-concurrency consumption data, from GaussDB (for Redis)
Read the data in.
- Offline training:
GaussDB (for Redis) stores the latest feature data, and OBS stores the full amount of feature data.
1) For simpler models that use static features, you can directly obtain features from OBS and use them.
2) For scenarios that use real-time features (such as a real-time recommendation system), Flink obtains user request records from Kafka in real time, and obtains features from GaussDB (for Redis) queries, stitches the records and features into training samples, and stores them in files , For offline training.
Feature platform's core demands for GaussDB (for Redis)
Combining the above business scenarios, summarize the core requirements of the feature platform for GaussDB (for Redis) as follows:
The key solution for GaussDB (for Redis) to meet the demands of characteristic platforms
1) Business interface
GaussDB (for Redis) is compatible with the community Redis 5.0 interface, supports the docking with Spark/Flink Connector, and satisfies the needs of the business.
2) Stability
GaussDB (for Redis) adopts a self-developed kernel to solve the old difficult problems of community Redis such as fork and oom, and has the stability of enterprise-level applications
3) Reliability
Zero data loss: order real-time disk placement, three copies of the bottom layer redundant storage, no risk of data loss
Strong data consistency: Based on GaussDB's common shared storage component DFV, three copies are strongly consistent, and there is no risk of dirty reads for multi-point access
4) Cost
GaussDB (for Redis) realizes automatic cold and hot separation of data, and adopts a hybrid storage solution of memory + SSD, which greatly reduces the customer's cost of use. According to VMALL's feature volume measurement, for 100 million users, the number of features for each user is K-number 10K, and the cost of GaussDB (for Redis) is only a little over 3W a year. If you choose community Redis, the cost is 20W+
5) Performance
GaussDB (for Redis) adopts a multi-threaded architecture, and all nodes can support writing at the same time, so it can better meet the high-throughput writing requirements of batch irrigation. In terms of reading, based on the cold and hot separation scheme, the resident memory of hot data provides stable and low latency; cold data reading involves IO exchange and has a certain long tail, but it can meet VMALL business requirements (currently VMALL online GaussDB (for Redis) instance reading The average delay is 0.16ms, P99 0.4ms, P9999 1.5ms)
6) Scalability
Based on the separation of computing and storage architecture, the underlying data can be accessed by any node, and no data copy and relocation occurs during the expansion process, so the speed is extremely fast; the expansion of the computing node is completed in minutes, and the storage expansion is completed in seconds, with RTO <10 seconds
In summary, compared with the community Redis, GaussDB (for Redis) provides a more stable user experience, more reliable data storage, lower cost of use and more convenient expansion capabilities. It is more suitable for large-scale platforms like the VMALL feature platform. Enterprise-level Redis service for e-commerce big data applications. Therefore, the VMALL feature platform finally chose GaussDB (for Redis) as the storage service for feature data.
4 Effect after going online
At present, VMALL has completed the first phase of feature data migration, including "Spark offline feature production" in the "feature production" business, and "offline training Flink feature query" in the "feature consumption" business, which has been migrated to GaussDB (for Redis ).
current GaussDB (for Redis) runs smoothly, and the delay is stable during peak business hours, which can meet the current business requirements of VMALL. Among them, the average read delay is 0.2ms (p99<0.4ms), and the average write delay is 0.6ms (P99<2ms).
VMALL has started the second phase of feature data migration, and plans to complete the access to core services including "Flink online feature generation" and "online reasoning".
5 Summary
This article introduces the selection and application of feature data storage services in the process of building a feature platform for Huawei Mall (VMALL). It can be seen that the Huawei Cloud GaussDB (for Redis) service has advantages in terms of cost, reliability, and scalability. It can be used as an ideal solution for feature data storage, providing enterprise-level stable and reliable Redis service capabilities.
Enterprise Redis special hot sale, unlimited new and old users, free for the first month (link):
Click to follow and learn about Huawei Cloud's fresh technology for the first time~
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。