Commodity 3D display is coming, HMS Core3D modeling service helps e-commerce development

The traditional e-commerce product display adopts the form of combination of pictures and texts, the copy introduces the relevant parameters of the product, and the exquisite pictures are used to attract the attention of customers. However, due to the color difference and size inconsistency of the graphic product display, consumers will have doubts about the inconsistency between the picture and the real object, and consumers need to consume a lot of energy to read and compare before making a purchase decision.

With the continuous improvement of Internet e-commerce platforms, short videos have gradually become the mainstream form of product display. By shooting short videos, products can be displayed in multiple directions, allowing consumers to understand the parameters and advantages of products in a short period of time, which greatly improves the shopping speed of consumers. However, the disadvantage of short video display is that the display time is short and it is difficult to interact with consumers.

The development of 3D technology has given the e-commerce industry the latest way to display products. 3D models can display products in 360° all-round details, allowing consumers to understand products more intuitively and enhance their confidence in products. Consumers can also interact with products through the screen, and can view products in 360 degrees by dragging with one finger, and freely zoom in to view high-definition details, thereby increasing the probability of ordering.

However, the current effective 3D modeling technology is prohibitive for the majority of developers due to its high cost.

1. High technical threshold: professionals plus professional equipment such as depth cameras.

2. High time cost: Professionals manually complete the production, rendering and adjustment of the model. Completing a low-precision model of a simple object starts in hours, while a high-precision model takes longer.

3. High cost: The professional modeling cost of a single product is high, with an average price of thousands of yuan, and complex models are even more expensive.

HMS Core's 3D modeling service for easy modeling. Users only need to use ordinary RGB cameras to automatically generate 3D geometric models and textures of objects by taking images of objects from different angles. For example, in the scene of physical display in e-commerce, you can use this ability to automatically generate the products you want to display. The model is used for 3D display. Users can zoom in or out of products in 360°, view product details, and provide users with a differentiated purchasing experience.

The 3D object modeling capability is completed by the end-cloud collaboration. The end-side is responsible for collecting RGB images, and shooting multiple images around the object to obtain images of different angles of the object. After shooting, upload it to the cloud to realize 3D object modeling. The process and key technologies of cloud modeling include target detection and segmentation, feature detection and matching, sparse point cloud computing, dense point cloud computing, and texture reconstruction.

Show results

Taking bread as an example, simply take multiple images around the bread to get a realistic 3D model of the bread. How is this function achieved? The following are the detailed development steps.

Preparation before development

1. Configure the integrated SDK package

In the build.gradle file of the application, add the SDK dependencies of the 3D modeling service in dependencies

 // 3D Modeling Kit SDK
implementation 'com.huawei.hms:modeling3d-object-reconstruct:1.0.0.300'

2. Configure AndroidManifest.xml

Open the AndroidManifest.xml file in the main folder, you can configure read and write mobile phone storage and camera permissions according to the scene and usage needs, add it before <application>

 <!-- 往sdcard中写入数据的权限 -->
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<!-- 使用相机的权限 -->
<uses-permission android:name="android.permission.CAMERA" />

Development steps

1. To use the cloud-side service capability, you need to use the api_key value in "agconnect-services.json", and set the application authentication information through api_key or AccessToken during application initialization. AccessToken has a higher priority. Both methods can be initialized and set once when the application starts, and there is no need to set it multiple times.

(1) Set the AccessToken through the setAccessToken method.

 ReconstructApplication.getInstance().setAccessToken("your AccessToken");

(2) Set api_key through the setApiKey method. When you register your app on AppGallery Connect, your app is assigned an api_key.

 ReconstructApplication.getInstance().setApiKey("your api_key");

2. Create a new 3D object modeling engine and initialize it, and create a new 3D object modeling configurator.

 // 新建3D物体建模引擎。
Modeling3dReconstructEngine modeling3dReconstructEngine = Modeling3dReconstructEngine.getInstance(context);
// 新建3D物体建模配置器。
Modeling3dReconstructSetting setting = new Modeling3dReconstructSetting.Factory()
    // 设置工作模式为图片模式。
    .setReconstructMode(Modeling3dReconstructConstants.ReconstructMode.PICTURE)
    // 设置贴图模式为普通模式或PBR模式。
    .setTextureMode(Modeling3dReconstructConstants.TextureMode.PBR)
    .create();

3. Create a new upload listener callback, which is used to process the upload result of the captured object image.

 private Modeling3dReconstructUploadListener uploadListener = new Modeling3dReconstructUploadListener() {
    @Override
    public void onUploadProgress(String taskId, double progress, Object ext) {
         // 上传进度。
    }
    @Override
    public void onResult(String taskId, Modeling3dReconstructUploadResult result, Object ext) {
         // 上传成功处理。
    }
    @Override
    public void onError(String taskId, int errorCode, String message) {
        // 上传失败处理。
    }
};

4. Use the 3D object modeling configurator to initialize the task, and set the upload listener for the newly created 3D object modeling engine to upload the collected image data.

 // 使用3D物体建模配置器初始化任务（该接口需要在子线程中调用）。
Modeling3dReconstructInitResult modeling3dReconstructInitResult = modeling3dReconstructEngine.initTask(setting);
String taskId = modeling3dReconstructInitResult.getTaskId();
// 设置上传监听器。
modeling3dReconstructEngine.setReconstructUploadListener(uploadListener);
// 调用3D建模引擎的上传接口，上传采集的图片数据。
modeling3dReconstructEngine.uploadFile(taskId, filePath);

5. Query the status of the 3D object modeling task.

 // 查询3D物体建模任务状态需要初始化任务处理类。
Modeling3dReconstructTaskUtils modeling3dReconstructTaskUtils = Modeling3dReconstructTaskUtils.getInstance(context);
// 调用查询接口获取3D物体建模任务状态（该接口需要在子线程中调用）。
Modeling3dReconstructQueryResult queryResult = modeling3dReconstructTaskUtils.queryTask(taskId);
// 获取建模任务状态。
int status = queryResult.getStatus();

6. Create a new listener callback, call the preview function, and preview the 3D model

 Modeling3dReconstructPreviewListener previewListener = new Modeling3dReconstructPreviewListener() {
    @Override
    public void onResult(String taskId, Object ext) {
        // 3D物体建模预览结果。
    }
    @Override
    public void onError(String taskId, int errorCode, String message) {
        // 预览错误回调函数。
    }
};
// 预览模型配置。
Modeling3dReconstructPreviewConfig config = new Modeling3dReconstructPreviewConfig.Factory().setTextureMode(Modeling3dReconstructConstants.TextureMode.PBR).create();
// 预览模型。
modeling3dReconstructEngine.previewModelWithConfig(taskId, context,config, previewListener);

7. Create a new download listener callback to process the download result of the 3D object modeling model file.

 private Modeling3dReconstructDownloadListener modeling3dReconstructDownloadListener = new Modeling3dReconstructDownloadListener() {
    @Override
    public void onDownloadProgress(String taskId, double progress, Object ext) {
        // 下载进度。
    }    
    @Override
    public void onResult(String taskId, Modeling3dReconstructDownloadResult result, Object ext) {
        // 下载成功处理。
    }
    @Override
    public void onError(String taskId, int errorCode, String message) {
        // 下载失败处理。
    }
};

8. The newly created download configuration item and the newly created download listener are passed to the newly created 3D object modeling engine, and the reconstructed model file is downloaded.

 // 设置下载配置项。
Modeling3dReconstructDownloadConfig downloadConfig = new Modeling3dReconstructDownloadConfig.Factory()
    // 配置OBJ或glTF格式。
    .setModelFormat(Modeling3dReconstructConstants.ModelFormat.OBJ)
    // 配置普通或PBR模式。
    .setTextureMode(Modeling3dReconstructConstants.TextureMode.PBR)
    .create();
// 设置下载监听器。
modeling3dReconstructEngine.setReconstructDownloadListener(modeling3dReconstructDownloadListener);
// 调用3D物体建模引擎的下载接口，传入任务id，下载地址和下载配置项，下载模型文件。
modeling3dReconstructEngine.downloadModelWithConfig(taskId, savePath, downloadConfig);

After completing the above steps, 3D modeling capabilities can be realized in e-commerce applications, and 3D models can be quickly created for products with mobile phones. Interested developers can experience it immediately!

In addition to product 3D modeling and display, the HMS Core e-commerce solution also provides AR interactive shopping capabilities, which can realize the function of product try-on and create a shopping experience that integrates virtual and reality for consumers. The relevant content will be detailed in the next issue. Explanation, please look forward to it.

Learn more details>>

Visit the official website of Huawei Developer Alliance
Get development guidance documents
Huawei Mobile Services Open Source Warehouse Address: GitHub , Gitee

Follow us to know the latest technical information of HMS Core for the first time~