This article will take you GNN from entry to take-off, and be the most stable GNN fan!

Abstract: This article introduces the development of graph neural networks in academia and industry, and gives the basic concepts and representations of graph neural networks, summarizes the variants of graph neural networks, and finally introduces the framework of Huawei cloud graph neural network.

This article is shared from the HUAWEI CLOUD community " , the soul of the meal, the soul of the meal, understand the graph neural network to stabilize the rice ", the original author: agile Xiaozhi.

Dry food! Don’t you understand the graph neural network (GNN) used by major manufacturers in 2021? This article will take you GNN from entry to take-off, and be the most stable GNN fan!

What exactly is a graph neural network?

To understand graph neural networks, you must first understand what graphs are! This kind of structure generally exists in human social life. For example, the social network between people will constitute a diagram, the subway line and the high-speed rail line will constitute a diagram, the purchase of goods by netizens will constitute a "netizen-commodity" diagram, and the web pages in the Internet The mutual links of, will constitute the graph, and the mutual reference of the paper will also constitute the graph. Many tasks can be accomplished based on the information in these graphs, such as predicting whether a user will buy or be interested in a product based on the historical interaction between the user and the product; another example is predicting the relationship between the user and the user based on the friendship or communication records between the users. Whether they constitute a friendship relationship.

Since graphs are everywhere, how do we analyze graphs? Currently, graph analysis tasks are usually solved by graph embedding technology. Through graph embedding technology, the structure and content of the graph can be represented by a low-dimensional vector, which can be used as input for downstream learning tasks. In addition, graph embedding can also be combined with deep learning technology. For example, based on the assumption of local connection and translation invariance, graph embedding can be combined with Convolutional Neural Network (CNN) to obtain Graph neural network (GNN).

The learning tasks on the map include:

1. Graph node classification task: each node in the graph has corresponding characteristics. When we know the category of some nodes, we can classify nodes of unknown type.
2. Graph edge structure prediction task: There are also many types of edge relationships between nodes in the graph. This task is to predict the relationship between nodes and nodes.
3. Classification of graphs: This task is to classify the entire graph. The basic idea is to aggregate the features of the nodes in the graph as the features of the graph, and then classify them.

How has the academic industry developed?

Academic development

In recent years, graph neural networks have ushered in a period of rapid outbreak. In terms of theoretical research, related work has conducted research on the principle explanation of graph neural networks, variant models, and the expansion and adaptation of various graph data. Counting the relevant papers at the top conferences in the past year, we can find that graph neural networks have become the biggest research hotspot.

Figure 2.1 The development of academic circles

Industry development

In terms of application and practice, graph neural networks show unprecedented penetration, ranging from visual reasoning to open reading comprehension problems, from drug molecule development to 5G chip design, from traffic flow prediction to 3D point cloud data learning, you can see The graph neural network has extremely broad application prospects.

Ant Financial uses the graph neural network model to mine the relationship pattern between "normal users" and "fraud gangs" in the equipment sharing graph, thereby realizing the identification of malicious accounts. Each node in the graph has its own characteristic information. Through these characteristic information, the device node information associated with a certain user node can be mined. When a certain user is associated with many devices, it can be considered that the user is at high risk. At the same time, based on the association relationship in the figure, the nodes connected to the malicious user and associated devices may also have a high risk.

Figure 2.2 Ant Financial: High-risk account identification

Didi Chuxing has studied a forecasting model of online ride-hailing demand based on spatio-temporal multi-graph convolutional neural network. By analyzing the complex temporal and spatial dependencies between regions, we can accurately predict the demand for online car-hailing, guide vehicle scheduling, improve vehicle utilization, reduce waiting time, and relieve traffic congestion to a certain extent.

Figure 2.3 Didi Travel: Vehicle regulation and management

Alimama uses the graph neural network to mine multiple relationships among Query (query words), Item (commodity) and Ad (advertising) from different dimensions such as user behavior logs and content attributes. For online request scenarios, efficient vectorized nearest neighbor retrieval is performed by calculating the distance between the user's query word vector, the node vector in the pre-behavior and the advertisement node vector, so as to quickly match the advertisement that meets the user's intention and recommend it to the user.

Figure 2.4 Alimama: search ad matching

NetEase Music uses the graph neural network to mine user characteristics, song characteristics, and user behavior characteristics of songs to achieve accurate music recommendation. Each node in the graph has structural information. If a user frequently subscribes to a certain category of music or has a higher score for a certain category of music, then the system can determine that the user is interested in this type of music, and can recommend more to the user. More music in this category.

Figure 2.5 NetEase: Music recommendation

Graph neural network

Basic concepts of graph theory

In response to the problem of non-Euclidean structured data representation, researchers have introduced graphs in the abstract sense of graph theory to represent non-Euclidean structured data.

Graph G is composed of a set of vertices and a set of connected edges, and can usually be defined as the following form:

The vertex set (Vertex) V can be expressed as

Edge set (Edge) E can be expressed as

Graph representation

The degree matrix, adjacency matrix and Laplacian matrix of the vertices are often used to describe the graph.

The degree matrix of the vertex (Degree) D: the number of edges associated with the vertex

Adjacency matrix (Adjacency matrix) A: Commonly used representation method of graph structure

Laplacian matrix L: a representation method of graph structure

The following figure shows examples of connected graphs and the corresponding degree matrix, adjacency matrix, and Laplacian matrix.

Figure 3.1 Basic concepts of graph theory

Graph neural network model

Graph embedding model

Graph Embedding (Graph Embedding) means that the nodes, edges or subgraphs in the graph are represented by low-dimensional continuous vectors. In order to obtain the graph embedding representation, the message propagation mechanism in the graph can be used. The message propagation mechanism in the figure includes two steps: message aggregation (aggregation/combine) and node update (update). Message aggregation refers to the characteristics of surrounding neighbor nodes, and node update refers to the embedding representation of learning center nodes. The message dissemination mechanism in the figure can be characterized by the following formula.

In the formula, □ represents a function that is derivable and has nothing to do with the input order, such as sum, mean or maximum function, etc.; γ and φ represent derivable functions, such as multilayer perceptrons.

Figure 3.2 Graph embedding model

Graph Convolutional Neural Network

In the graph convolutional neural network, the propagation mode between layers is:

The following figure is a schematic diagram of the graph convolutional neural network. The input of the graph convolutional neural network is a graph. After several layers, the node feature changes from X to Z, sharing the A parameters in the multiple hidden layers in the middle.

Figure 3.3 Graph Convolutional Neural Network

Construct a two-layer graph convolutional neural network. The activation functions adopt ReLU and Softmax respectively, and the overall forward propagation formula is:

Finally, according to the feature Z, downstream tasks can be done, such as node classification tasks, graph classification tasks, and node connection prediction tasks.

Graph attention network

The attention mechanism can be understood as a weighted summation process: for a given query, there are a series of values and one-to-one keys, so how to calculate the result of the query? It's very simple. Calculate the similarity between the query and all the keys, and then add weighted sums of all values according to the similarity. This similarity is the attention coefficients, and the calculation formula is as follows:

In the formula, a is the weight coefficient of the feedforward neural network, and || represents the splicing operation.

Figure 3.4 Graph attention network

Using the attention mechanism, the characteristics of each node in the graph can be updated:

Using the multi-head attention mechanism, K weight coefficients can be used to update the node features separately:

Figure 3.5 Multi-head mechanism in graph attention network

The advantages of graph attention networks include: parallel computing can be performed on different nodes, nodes with different degrees can be processed at the same time, and graph structures that have never been seen before can be processed and used to solve inductive learning problems.

Heterogeneous graph attention network

GCN operates directly on the homogenous graph, and induces fusion according to the attributes of its neighborhood to obtain the embedding representation of the current node. In a homogeneous graph, the propagation rules of each layer are shown in the following formula

In a heterogeneous network, there are many types of nodes T={τ1, τ2, τ3, …}, and GCN cannot be directly applied to a heterogeneous network. In order to solve this problem, we can use heterogeneous graph convolution, consider the heterogeneity of various types of information, and use type-related transformation matrices to project them into the public implicit space.

When a particular node is given, different types of neighboring nodes may have different effects on it. For example, neighboring nodes of the same type may carry more useful information, and different neighboring nodes of the same type may also have different effects. The importance of. Therefore, a two-layer attention mechanism of a heterogeneous network can be designed.

Graph Neural Network Platform

The difficulty of the current graph neural network platform development lies in the lack of a unified algorithm framework, and the need to improve the efficiency of data processing. The traversal of graph data and its interaction with deep learning will greatly reduce the computational efficiency of graphs, which is also one of the bottlenecks that graph deep learning has not been able to land. If you want to make a breakthrough in performance, you need to redesign a new graph deep learning framework. The following describes the Huawei Cloud Graph Neural Network Framework.

Figure 4.1 Multi-head mechanism in graph attention network

(1) GNN new framework based on graph engine: Based on the efficient neural network training operator in ModelArts, combined with the existing high-performance graph computing framework platform capabilities of Graph Engine Service (GES), the graph engine is used With the characteristics of high concurrency and low latency, the training process of GNN is highly parallelized, such as edge jump probability estimation, vertex neighborhood sampling, negative sample construction, etc., are all resolved into local operations for each vertex; the system provides With the dynamic scheduler, these local operations can be executed in a highly parallel manner, which can greatly improve the overall throughput of the system.

(2) Unification of multiple GNN algorithm frameworks: A unified architecture is used to achieve unsupervised large-scale graph embedding (such as DeepWalk, Node2Vec) and semi-supervised graph convolution (such as GCN, GraphSAGE) and other types of GNN algorithms, which reduces The maintenance cost of the system.

(3) GNN and graph data management integration: Enterprise-level GNN applications are usually not a one-time calculation, and the data scale is also very large, so these data must be maintained and managed. The existing GNN usually does not have this ability, users can only build another database for maintenance, and then export the data as a whole during calculation. It not only consumes resources, but also introduces many problems such as data consistency. GES adopts the property graph data model (Property Graph) and the ecologically compatible de facto standard Gremlin graph query language for distributed graph data management and maintenance. When training is needed, various operators are called locally in the graph engine and executed concurrently to reduce This reduces the end-to-end performance loss.

With the help of ModelArts' high-efficiency neural network training advantages and GES's high-performance graph computing advantages, Huawei Cloud Map Neural Network has greatly improved the overall computational efficiency of GNN. Taking the node2vec algorithm as an example, on the PPI data set, Huawei Cloud Map Neural Network samples The training can be completed within 2 minutes, which is 20 times higher than the traditional open source implementation.

Graph neural networks have a future!

As the popularity of graph neural network research continues to rise, different variants of graph neural network also continue to emerge. In addition, because graph neural networks have good expressive capabilities for non-European spatial data, they have broad application prospects in the intersection of e-commerce, finance, transportation, and social sciences with a large amount of data accumulation. This article introduces the development of graph neural networks in academia and industry, and gives the basic concepts and representations of graph neural networks, summarizes the variants of graph neural networks, and finally introduces the framework of Huawei Cloud Graph Neural Network. I hope this article can provide some reference for you on the road of GNN!

Click to follow and learn about Huawei Cloud's fresh technology for the first time~