Aurora Notes | Phase I Construction of Data Service Platform

background
Aurora currently has many business lines, and each business line has data service API development requirements. In the past, the company did not have a unified data service bus, which led to repeated development of data sources and data application APIs, and serious waste of resources. Data service platform The main purpose is to:
 Provide a unified internal and external data API interface, standardize the development and management of data service API
Combined with data map, open up data application and data engineering, realize the supplement of data value and link blood
1. Introduction to Aurora Data Service Platform
Jiguang Data Service Platform provides the ability to generate APIs from data tables, supports visual wizard mode or script mode to quickly develop API interfaces, supports relational databases and NoSQL databases, and can be used by internal and external systems to obtain data by calling API interfaces. It is open to API Perform unified management and release.

【core function】:
[Visual API Development] Provides a visual development API function, you can quickly configure an API to improve delivery efficiency
[Visual API debugging] Provide the most basic visual API debugging function, reducing test costs
[Unified API Management] Provide unified internal and external API management functions for each business line
[API Monitoring Alarm] Provides functions such as interface call statistics, visual reports, monitoring, and alarms
[API market] The APIs configured on the platform are released to the API market in a unified manner. Data development/product managers of each business can check whether the APIs released by the API market can create value for their own business, and can apply for use according to business needs

【user target audience】:
Data development: configure data source, configure API interface for rapid development of product interface/business interface
Data service platform development: management business interface, management data source, platform stability guarantee
Product operation: business interface monitoring, data analysis of product interface usage

1.1 Use process
API publisher process:

Figure 1-1-1
API user process:

Figure 1-1-2
1.2 Data source management
The data service platform provides data source management functions. The data development project registers the developed data tables on this platform, and business departments can only manage their own data sources.

Figure 1-2-1
The first-phase data source only supports pika and redis components, and more storage components (hbase/mysql/es, etc.) will be supported in the future
1.3 API development
With a data source, business development can configure API interface input and output information based on the data source, quickly generate and publish APIs, and improve service delivery efficiency.

Figure 1-3-1
The API developed through data services can standardize the definition of API interfaces and manage the API interfaces of various business lines in a unified manner.
1.4 API debugging
After configuring the API, it enters the API debugging stage, where business development can enter the value of the request parameter to call, view the request details and return content, and verify whether the API interface input and output parameters meet expectations.

Figure 1-4-1
1.5 API release
After the API debugging is passed, you can publish the API. Here you need to go through the work order approval. After the approval is passed, the API gateway will be released automatically.
1.6 API Gateway
As a data service API gateway, it must have functions such as identity authentication, authorization verification, and frequency and current limitation.
Identity authentication: In order to solve the interface security problem, a pair of devKey and devSecret will be assigned to each product line. When the API interface is called, it is necessary to bring the devKey and signature information to access.
Authorization verification: Each published API must be authorized by the person in charge of the API to access it.
 Frequency limit and current limit: The person in charge of the API can set the QPS and QPD of the interface when publishing the API. If the threshold is exceeded, the access to the interface will be restricted.
1.7 API market
After the API is successfully released, it will be put on the API market. Business development can search and view the input parameters, output parameters, error codes, publishers and other information of the API interfaces that have been listed in the API market. You can also apply for certain API permissions and approvals. After passing, you can directly call the API interface to obtain data.

Figure 1-7-1

Figure 1-7-2
1.8 Service overview
The platform has a service overview function, including statistical functions such as the number of published APIs, the number of unpublished APIs, the number of successful calls to APIs, the number of failed calls to APIs, and the distribution of error codes, as well as the view of edited and unpublished (draft status) API lists.

Figure 1-8-1
2. Aurora Data Service Platform Architecture Design

2.1 Product architecture diagram

Figure 2-1-1
From the product level, the data service platform is mainly divided into four layers:
1. Application layer: The business can apply to call certain APIs to obtain data based on needs.
2. Function layer: The platform provides functions such as API creation -> API debugging -> API release -> API monitoring.
3. Support layer: some basic functions used by the support platform, including: user management, role management, log management, work order approval and other functions.
4. Data source layer: data storage components, including: pika, redis, hbase, mysql, es, etc.

2.2 Technical architecture diagram

Figure 2-2-1
The data service platform is mainly divided into two parts:
[Management side]:
The management terminal is mainly provided for business development. Through the management terminal, business development can quickly complete operations such as configuring data source -> developing API -> publishing API.
【Server】:
The server provides internal or external API interface access, which is divided into three layers:
Gateway layer: API call entrance, mainly responsible for authentication, authority, API routing, frequency limitation, current limitation, etc.
Interface layer: Provide API interface query service, assemble parameters according to API information and return query results.
Data access layer: This layer provides access to business data storage components.
2.3 Overall interaction diagram

Figure 2-3-1
After the API publisher generates and publishes the API through the management terminal, the API interface metadata information (data source, input parameters, output parameters, QPS, QPD, etc.) will be stored in redis for use by certification centers, gateways, and configurable services.
2.4 Configurable interface service
In the first phase of the construction of the data service platform, Pika and redis configurable (NoSQL API) interface capabilities are provided. The data source is connected to pika and redis through the jcache proxy layer. Business data is stored in KV mode, which can be stored in a simple key- Value provides external services, using key as an input parameter and value as an output parameter to abstract the API interface provided externally.
The configurable interface service provides an abstract interface to the outside. In this abstract interface, the metadata information (data source, input parameter, output parameter, etc.) of the interface is obtained according to the interface id, and then the data source connection is created according to the interface metadata information to generate storage key, get the value value, and finally encapsulate the parameters and return.

Figure 2-4-1
After the user identity authentication, api call permission authentication, current limit and other pre-logics are passed, api forwarding is performed. Any path api (Jcache configurable interface) created and published on the data service platform will be forwarded to the abstract api of the configurable interface service through the gateway.

3. Follow-up planning
Although the first-phase functions have been launched, they can only meet part of the business needs. There are still many functions that need to be improved and developed. The following are the second-phase functions:
The data source supports more storage components, such as hbase, mysql, es, hive, etc.
Increase the pre-processing and post-processing capabilities of the configuration API
Achieve API service orchestration function, support service serial, parallel, branch and other capabilities
 Increase the elastic scaling and resource isolation functions of the API interface
Add the registration API function, the business can register the existing API to the data service platform
Based on the product functions of the first phase, combined with business needs, perfect platform functions