Application of visualization technology in Nebula Graph

This article was first published on the Nebula Graph Community public account

This article is organized from the #visualizationonLive 🔗 theme live broadcast . In this live broadcast, 3 visualization guests talked about the visualization in their eyes and the "pits" they stepped on in the practice of visualization projects.

• Lu Xiaolong: Product leader of Nebula Explorer, a visualization tool, visualization technology expert;
• Miao Zhuang: Core development of Nebula Explorer, a visualization tool, visualization technology expert;
• Wang Yang: product manager of Nebula Dashboard, a visual monitoring operation and maintenance tool, and visualization technology expert;

What is visualization?

From a personal understanding, visualization is to display some relatively abstract things we usually see, such as icons or charts, in a reasonable way, making it more convenient for users to interact. To put it simply, the common buttons and icons on web pages, such as entering: https://www.baidu.com/ and the blue "Baidu click" button, and the accessibility icons in the lower right corner are all simple and basic Visual display. Without these small icons, you may need to call the terminal to execute a command to conveniently complete the corresponding operation. Visualization can greatly reduce operating costs and make it easier for users to understand corresponding operations.

In the above example, we are talking about UI visualization. When it comes to UI, let's read the memory of more than ten years ago. Everyone knows that in the early years, computer technology was not yet developed, and everyone looked at the command line. And now, why can computers fly into ordinary people's homes? It is because the computer or the computer has an interface, a UI, and something in line with human intuition. Therefore, the visual thing that can improve efficiency is called visualization. If you want to give a definition, there are probably two points in visualization:

1. The above example and GUI, this kind of visualization is visible to everyone, but in my opinion, it is not the biggest highlight of visualization;
2. Design, or fine arts, is the soul of visualization. Because of this thing, it gives meaning to the visualization. In layman's terms, whether a thing is useful or not, it is often not easy to use if you can make it; but if it has good interaction, good experience, and good visual effects, it can conform to human intuition and dig out some information. Even if it is a good visualization. Therefore, I think that visualization still needs two parts, one part is in the physical sense, that is, the rendering of the first part, and the second part is its design experience, which involves hardware and software.

In fact, the concept of visualization can be divided into two dimensions: broad and narrow.

In a broad sense, visualization can be understood as processing a complex piece of information to make it easier and more intuitive to present to the user, allowing users to accept some information more conveniently. This belongs to the category of visualization, like GUI, in fact, it is a kind of visualization, which abstracts the underlying commands in a visual way, so that users can interact with the machine more conveniently.

In a narrow sense, it is the application for the scene. For example, with the development of the Internet, the amount of data we deal with will become larger and larger. We need to present this data in a certain way. The better the presentation, the better the data expression and the more understandable it is. How to display more data more smoothly on the basis of certain performance bottlenecks in current hardware may be the meaning of current visualization. From this perspective, from a narrow perspective, visualization may be charts, Canvas animations, or the rise of VR technology in the metaverse, browser rendering, and 3D rendering. These visualizations can be researched.

Specifically, because Nebula is a distributed graph database, the visualization we mentioned is more of a data visualization. For example, Nebula Dashboard is a visual operation and maintenance monitoring product. The visualization practice in this area is not only about icons and animations, but also how to use complex operation and maintenance information in a simple way to make it easier for users to control the cluster.

visualization technology

At different stages and under different scenarios, we will choose different visualization technologies to better serve products and businesses.

As far as visualization technology is concerned, the common visual rendering technologies for front-end display are SVG, Canvas, CSS and WebGL, and what to use to draw SVG and Canvas is another topic. D3, which is often mentioned by the Nebula community, is not actually a visualization technology. In essence, it is a visualization algorithm tool that can be used on web pages or in terminal applications. D3 itself has some peripheral tools to do simple SVG encapsulation, which is more upper-level application tools, such as the chart library ECharts.

Wang Yang, product manager of Nebula Dashboard, said that common visual rendering technologies such as CSS and SVG as mentioned above will be customized and developed according to the application scenarios and computing resources faced. For example, in order to facilitate third-party access, the designers of the visualization library will abstract the syntax and make the syntactic sugar more scientific and standardized; in scenarios with high performance requirements, they will encapsulate the tools to achieve high performance. Here he mentioned Ant's open-source visualization engine: G2 https://github.com/antvis/g2/ , G2 was inspired by the concept of the book "The Grammar of Graphics", and designed this visual underlying engine based on graphics grammar theory , G2's grammar implementation is also a scientific standard recognized by the industry. Compared with other visualization libraries, G2 is more like a complete product in terms of syntax design.

Speaking of G2, Miao Zhuang, the core developer of Nebula Explorer, a visual graph exploration tool, said that although G2 grammar is standardized and scientific, it is not a very convenient visualization tool if you really want to do data visualization, because it is based on the theory of graph grammar and requires a certain degree of professionalism. knowledge, so there will be a certain threshold. More often, G2 may still be used as a visual charting library rather than a visualization library.

Visualize performance issues

Nebula Explorer is a graph exploration tool, and a single page will face the problem of node data display. Users in the community often ask how many node data can be rendered on a single canvas page. In response to the visualization performance problem, core product developer Miao Zhuang said: visualization performance is not an absolute performance, it is often accompanied by user experience. The user experience and performance are inversely proportional to some extent. The better the performance, the better the user experience is, and the information presented will be relatively less; and the more data presented, the more hardware resources will be consumed. This is a question of balance.

There are two ways to improve the visualization performance: one is to improve the hardware configuration, which was a popular stalk before - is distributed concurrency not enough? Add a machine, there is nothing that cannot be solved by adding a machine. However, the simple and rude method of adding machines is not a good idea. Although it solves the physical limitations, it adds the cost problem. In addition to the machine cost, there is also the corresponding system maintenance cost. The addition of one machine or multiple machines will inevitably increase the complexity of the system. In addition, adding machines is not a universal solution, and some users may not be able to add machines at will. The other is the optimization algorithm. Compared with adding a machine, the second method is more difficult. The current algorithm itself is perfect, and further optimization is only a marginal diminishing effect.

Optimization practice of Nebula visualization tool

The optimized performance of Nebula Explorer is also developed from the above two software and hardware directions. In terms of hardware, the hardware performance of different users is different. During the development process of Explorer, better hardware will be used to realize the presentation of cool special effects; while for users with poor hardware performance, two modes of high image quality and normal image quality are provided. , adaptively adapt to the user's hardware to achieve a better rendering effect. Although Explorer's 3D image exploration mode looks cool and "eats" computing resources in the past, the hardware requirements of this function are not as high as imagined, and any mobile phone's GPU configuration can run Explorer 3D mode, relatively The CPU performance of the language also determines the user experience.

In fact, in addition to performance optimization on software and hardware, there is also strategy optimization on product design. Like some more difficult performance problems to solve, you can try to avoid it. In terms of performance optimization, there is an eternal principle, which is to ensure that 60 frames are rendered in 1 second. Even in special scenarios, 60 frames per second cannot be guaranteed, but more than 40 frames can ensure the smoothness of the system. This is like an optimized implementation of React. It disassembles many large tasks into multiple small tasks, and then inserts small tasks when the system is idle. There is another way of thinking. If it is a problem that cannot be solved at the code level, the corresponding interaction method can be modified. For example, a long time-consuming task is running. If the page is not clickable at this time, you will feel very stuck, right? However, adding a loading animation at this time will greatly reduce the user's waiting anxiety and make the user feel less stuck.

Here, let me tell you a personal experience: the performance pit that Nebula Dashboard stepped on when it was first launched to visualize the big screen. When the visual large screen was first launched, I just thought that a simple large screen might not involve too many performance problems. As a result, as soon as the CPU went online, I was stunned: the Nebula Dashboard visualization screen has only 7 or 8 charts. Although the interface is cool, in fact, the entire network request is only a dozen or so, so the problem that the CPU is requesting computing resources is too high. In addition, when the big screen went online, there were not many concurrency situations... After a round of investigation, the final positioning was that SVG consumed excessive resources. So we adjusted the animation implementation scheme: animations with high CPU requirements are downgraded to reduce the number of renderings and rendering frequency. This is a bit like image compression. If you drop a 1 MB image to a compressed website, it may be 200 KB after compression. At this time, the user is actually invisible to the naked eye.

Visualization Technology Selection

As mentioned above, hardware and software can improve visualization performance, but whether the technical selection is good or not can also affect the performance of your system.

Before the version of Nebula Explorer v2.2, due to technical selection and business logic organization problems, once the display problem of hundreds of nodes was encountered, it would encounter a freeze, and the animation was obviously not smooth. After that, Nebula Explorer was refactored to replace the G6 rendering engine at that time. G6 is a set of graph visualization engines based on G2. It may be because G6 is too focused on business scenarios and lacks the verification of high-performance scenarios, its performance in Nebula Explorer is not satisfactory. After switching to Force Graph, the performance improved by about 10 times. Of course, it is not that the performance of G6 is not good, but the visualization requirements of Nebula Explorer and G6 do not match.

Contrary to Nebula Explorer, Nebula Dashboard does not have thousands of point data displays, but a relatively fixed number of chart displays and corresponding operation and maintenance operations. For monitoring charts, Grafana is currently recognized by the market, and Grafana itself is an open source product. So, we went to study the implementation of Grafana and found that it uses the lightweight uPlot at the bottom https://github.com/leeoniya/uPlot . Because uPlot is less than 40kb, it is characterized by drawing time series diagrams, which is convenient for customized development. In addition to monitoring, Nebula Dashboard also has a more important function for cluster management, operation and maintenance operations. This is a departure from Grafana's focus on monitoring. In addition, the three criteria for Nebula Dashboard selection are: first, easy development; second, active open source community; third, more basic libraries can improve development efficiency. Finally, G2 was chosen as the visualization underlying tool of Nebula Dashboard in order to improve development efficiency.

Visual product design

Nebula Explorer has two graph exploration modes, 2D and 3D, which involve how the page is laid out. In the design of Nebula Explorer, algorithms are used to achieve the aggregation and hashing of specific point groups; and the full-image bird's-eye view mode is displayed in 3D mode, because the mainstream computer is 1920*1080, and a plane can display up to one million pixels. Then at this time, from a different angle, we can present more data in 3D than in 2D.

Visual knowledge

About rendering

VR, monitoring indicators, multi-point display, and rendering graphics when calculating the underlying rendering are mostly the same. For example, some computer systems use OpenGL technology to render 3D graphics that are composed of several triangles, and the same is true for Nebula Explorer's computational rendering, that is, a spherical node you see is actually composed of several triangles. Triangles are constructed and linearly interpolated with different vertex colors. In addition to the computer's rasterization drawing method, the popular ray tracing rendering in recent years is also a rendering technology with good effect. This technology uses the physical simulation method of ray emission, reflection and loss to obtain the final color of the pixel. Visually it will be closer to the real world.

Exchange graph database technology? To join the Nebula exchange group, please fill in your Nebula business card first, and the Nebula assistant will pull you into the group~~