How do I model a doll and make it dance?

Everyone must have imagined such a scene when they were playing with toys when they were young: imagine that the toys in their hands can move, just like the toys in "Toy Story", they have their own characters and can play with themselves. I used to think that Toy Story was just a fairy tale, and the idea that toys had a soul and could move was just a fantasy. But now HMS Core 3D modeling service makes that fantasy a reality.

Principle explanation

Using the bone binding capability of HMS Core 3D modeling service , the doll can move on the mobile phone screen. Developers only need to use ordinary mobile phones to take multiple images of bipedal humanoid objects, build static 3D models, and automatically use cloud-side AI. After the binding algorithm, the skeleton and skin weights of the model are automatically generated, which can be driven by one-click through the mainstream engine in the industry. In addition to the preset actions, you can also add your own motion-captured personalized actions to let the doll figure dance to the rhythm on the screen.

Demo

Development steps of automatic skeletal binding ability

Before starting development work, you need to configure AppGallery Connect , and make sure that the project has been integrated with the HMS Core SDK and the Maven warehouse address has been configured.

1. To use the cloud-side service capability, you need to use the api_key value in "agconnect-services.json", and set the application authentication information through api_key or AccessToken during application initialization. AccessToken has a higher priority.

The AccessToken is set by the setAccessToken method, which can be initialized once when the application starts, and there is no need to set it multiple times.

 ReconstructApplication.getInstance().setAccessToken("your AccessToken");

To obtain the Access Token, see Open Authentication Client Mode Based on OAuth 2.0 .

The api_key is set by the setApiKey method, which can be initialized and set once when the application starts, and there is no need to set it multiple times.

 ReconstructApplication.getInstance().setApiKey("your api_key");

When you register your app on AppGallery Connect, your app is assigned an api_key.

2. Create a new 3D object modeling engine and initialize it, and create a new automatic bone binding configurator.

 // 新建3D物体建模引擎 
Modeling3dReconstructEngine modeling3dReconstructEngine = Modeling3dReconstructEngine.getInstance(context); 
// 新建骨骼绑定蒙皮配置器 
Modeling3dReconstructSetting setting = new Modeling3dReconstructSetting.Factory() 
    // 设置工作模式为图片模式 
    .setReconstructMode(Modeling3dReconstructConstants.ReconstructMode.PICTURE) 
    // 设置任务类型 
    .setTaskType(Modeling3dReconstructConstants.TaskType.AUTO_RIGGING) 
    .create();

3. Create a new upload listener callback to process the upload result of the captured object image

 private Modeling3dReconstructUploadListener uploadListener = new Modeling3dReconstructUploadListener() { 
    @Override 
    public void onUploadProgress(String taskId, double progress, Object ext) { 
         // 上传进度 
    } 
    @Override 
    public void onResult(String taskId, Modeling3dReconstructUploadResult result, Object ext) { 
         // 上传成功处理 
    } 
    @Override 
    public void onError(String taskId, int errorCode, String message) { 
        // 上传失败处理 
    } 
};

4. Use the 3D object modeling configurator to initialize the task, and set the upload listener for the newly created 3D object modeling engine to upload the collected image data.

 // 使用3D物体建模配置器初始化任务（该接口需要在子线程中调用） 
Modeling3dReconstructInitResult modeling3dReconstructInitResult = modeling3dReconstructEngine.initTask(setting); 
String taskId = modeling3dReconstructInitResult.getTaskId(); 
// 设置上传监听器 
modeling3dReconstructEngine.setReconstructUploadListener(uploadListener); 
// 调用3D建模引擎的上传接口，上传采集的图片数据 
modeling3dReconstructEngine.uploadFile(taskId, filePath);

5. Query the status of the automatic bone binding task

 // 查询3D物体建模任务状态需要初始化任务处理类 
Modeling3dReconstructTaskUtils modeling3dReconstructTaskUtils = Modeling3dReconstructTaskUtils.getInstance(context); 
// 调用查询接口获取3D物体建模任务状态（该接口需要在子线程中调用） 
Modeling3dReconstructQueryResult queryResult = modeling3dReconstructTaskUtils.queryTask(taskId); 
// 获取建模任务状态 
int status = queryResult.getStatus();

6. Create a new download listener callback to process the download result of the automatic skeleton binding model file

 private Modeling3dReconstructDownloadListener modeling3dReconstructDownloadListener = new Modeling3dReconstructDownloadListener() { 
    @Override 
    public void onDownloadProgress(String taskId, double progress, Object ext) { 
        // 下载进度 
    }     
    @Override 
    public void onResult(String taskId, Modeling3dReconstructDownloadResult result, Object ext) { 
        // 下载成功处理 
    } 
    @Override 
    public void onError(String taskId, int errorCode, String message) { 
        // 下载失败处理 
    } 
};

7. The newly created download configuration item and the newly created download listener are passed to the newly created 3D object modeling engine, and the model file with the successful skeleton binding is downloaded

 // 设置下载配置项 
Modeling3dReconstructDownloadConfig downloadConfig = new Modeling3dReconstructDownloadConfig.Factory() 
    // 配置OBJ或glTF格式 
    .setModelFormat(Modeling3dReconstructConstants.ModelFormat.OBJ) 
    // 配置普通或PBR模式 
    .setTextureMode(Modeling3dReconstructConstants.TextureMode.PBR) 
    .create(); 
// 设置下载监听器 
modeling3dReconstructEngine.setReconstructDownloadListener(modeling3dReconstructDownloadListener); 
// 调用3D物体建模引擎的下载接口，传入任务ID，下载地址和下载配置项，下载模型文件 
modeling3dReconstructEngine.downloadModelWithConfig(taskId, fileSavePath, downloadConfig);

More application scenarios

Using the bone binding function, in addition to making the doll dance on the phone screen, it can also help developers to do many other interesting and powerful functions. For example, in the field of e-commerce, the use of automatic skeletal binding can drive the dynamic display of the figure, thereby increasing the attractiveness of the product; in the field of education, the skeletal binding and audio functions can be combined, so that the dolls in the hands of children can dance and tell stories. Teaching is fun; in the field of games, skeleton binding and AR functions can be combined to allow 3D models to enter the real scene.

Learn more details>>

Visit the official website of Huawei Developer Alliance
Get development guidance documents
Huawei Mobile Services Open Source Warehouse Address: GitHub , Gitee

Follow us to know the latest technical information of HMS Core for the first time~