Online text entity extraction capability to help applications parse massive text data

With the development of informatization, a lot of valuable knowledge is hidden and distributed in massive data, which affects the efficiency of people's acquisition of knowledge. How to deal with complex unstructured text data has become a difficult problem.

Recently, HMS Core Machine Learning Service Version 6.5.0 added the online text entity extraction capability , which can detect whether there is entity information such as date, name, proper noun, etc. in the text, and extract such entities, that is, automatically. The ability to process unstructured natural language text data. For example, in the application of the film and television industry, there are often a lot of textual film reviews, information and other content. Using the online text entity extraction ability can quickly extract structural information, help build a knowledge map, and facilitate users to understand clearly.

In addition, the ability of text entity extraction is more used in question answering systems, information indexing, knowledge graph construction and other fields.

Question answering system

A question answering system is an advanced form of information retrieval system that can answer user questions in accurate and concise natural language. In the implementation process of the question answering system, it is necessary to use the text entity extraction ability to identify the question and entity information in the knowledge base, and then match the accurate answer through a variety of algorithm models.

Information index

Using online text entity extraction capabilities, specific entity information can be named as indexes and hyperlinks. For example, the proper nouns mentioned by users in comments can generate hyperlinks to facilitate other users to retrieve and understand related content.

Knowledge graph construction

The knowledge graph is a data structure composed of entities, relationships and attributes, that is, a knowledge base with a directed graph structure. For example, to construct a music knowledge graph, it is necessary to extract singers, songs, lyrics, movies and other related information from a large amount of text data first, and then build a knowledge graph.

At present, Huawei's machine learning service online text entity extraction capability supports a total of 16 entity categories, including names, money, film and television names, and web page links, which can be applied to different categories of apps according to actual semantic scenarios.

Integration steps

1. development preparation

For detailed preparation steps, please refer to the official website of Huawei Developer Alliance .

2. Integrate and configure apigateway authentication

Authentication mechanism based on apigateway:

 "paths": {
"/entityExtract": { "post": { "operationId": "entityExtract",
"parameters": [{"in": "body",  "name": "req",  "required": true,
"schema": { "$ref": "#/definitions/NerEnterReq"  } },  {
"name": "X-Request-ID", "in": "header",  "required": true,
"type": "string"
}, {"name": "X-Package-Name",  "in": "header", "required": true,
"type": "string"  }, ……],
 "responses": {"200": {   "description": "response of 200",
"schema": { "$ref": "#/definitions/ResponseEntityNerBodyVo"}}}}}

3. Create Online Text Entity Constructor

 // 使用自定义的参数配置创建语种检测器。
 MLRemoteNerSetting setting = new MLRemoteNerSetting.Factory()
                .setSourceLangCode("zh")
                .create();
MLRemoteNer ner = MLNerFactory.getInstance().getRemoteNer(setting);
进行文本实体抽取。

4. Get Text Entity Extraction Online

Asynchronous method example code:

 ner.asyncEntityExtract(input).addOnSuccessListener(new OnSuccessListener<RemoteNerResultItem[]>() {
            @Override
            public void onSuccess(RemoteNerResultItem[] remoteNerResults) {
                // 成功的处理逻辑。
                if(remoteNerResults != null){
                    // 有识别结果
                }else {
                    //  识别结果为空
                }
            }
        }).addOnFailureListener(new OnFailureListener() {
            @Override
            public void onFailure(Exception e) {
                // 识别失败,获取相关异常信息。
                try {
                    MLException mlException = (MLException) e;
                    // 获取错误码，开发者可以对错误码进行处理，根据错误码进行差异化的页面提示。
                    int errorCode = mlException.getErrCode();
                    // 获取报错信息，开发者可以结合错误码，快速定位问题。
                    String errorMessage = mlException.getMessage();
                } catch (Exception error) {
                    // 转换错误处理。
                }
            }
        });

Synchronous method sample code:

 try {
       RemoteNerResultItem[] remoteNerResults = ner.syncEntityExtract(input);
       // 识别成功逻辑
       if(remoteNerResults != null){
       // 有识别结果
       }else {
       //  识别结果为空
        }
   } catch (MLException mlException) {
       // 失败的处理逻辑。
       // 获取错误码，开发者可以对错误码进行处理，根据错误码进行差异化的页面提示。
       int errorCode = mlException.getErrCode();
       // 获取报错信息，开发者可以结合错误码，快速定位问题。
       String errorMessage = mlException.getMessage();
   }

5. When done, release resources

 if (ner != null) {
    ner.stop();
}

Learn more details>>

Visit the official website of Huawei Developer Alliance
Get development guidance documents
Huawei Mobile Services Open Source Warehouse Address: GitHub , Gitee

Follow us to know the latest technical information of HMS Core for the first time~