Intel OpenVINO™: One-stop "device-edge-cloud" for high-performance AI inference computing

In the context of the continuous driving of 5G communication, big data, artificial intelligence and other technologies, the coordinated development of cloud computing and edge computing is becoming an important trend in the future. With the continuous evolution of cloud-edge collaboration and the continuous large-scale deployment of AI solutions, the AI+IoT edge-cloud architecture has become the key to solving the problem of computing power in the digital transformation of enterprises and typical work scenarios.

In this process, Intel, which has made efforts to get through the adaptation of "software and hardware" and has made continuous innovations and breakthroughs in AI chips and AI computing power, took the lead in proposing and mastering the "four superpowers" that drive innovation and change - artificial intelligence, Ubiquitous computing, ubiquitous connectivity, interaction and collaboration between infrastructures from the cloud to the edge , and an even bigger revolution is being explored - Intel OpenVINO™ (version 2.2.1).

At the "Intel + Microsoft 2022 China AI Developer Summit" held on June 28, Wu Zhuo, an AI software evangelist from Intel, Joe Chen, an AI engineer from Microsoft's Artificial Intelligence and IoT Lab, and a group of experts brought us The new version of Intel OpenVINO™ and other related wonderful sharing.

The new Intel OpenVINO™ (version 2.2.1) drives cloud-side-end through efficient hardware to quickly realize high-performance artificial intelligence inference with "seven lines of code to complete inference" to solve the large-scale data computing needs of enterprises in complex business And simplify the cloud scale development and other problems, continue to accelerate AI reasoning, empower developers.

Use OpenVINO™ to quickly realize high-performance artificial intelligence inference in "cloud-edge-device"

As we all know, in the process of realizing the Internet of Things upgrade, the AI+IoT edge-cloud architecture solution can enable the data collected at the edge to achieve more modern and accelerated operations. The newly upgraded Intel OpenVINO™ (version 2.2.1) ( https://www.openvino.ai ) can not only help enterprises to realize large-scale complex business and large-scale data computing services in the cloud, but also support multiple computing services in the cloud. Different AI frameworks and services.

Using OpenVINO™ to realize the edge-cloud architecture has many advantages, which can simplify the training and development of the cloud, avoid the steps of downloading software, configuration and installation, and directly use all computing resources and services provided by the cloud; the edge can perform inference to obtain real-time decisions, thus Get inference results quickly, while avoiding the high cost of transferring large amounts of data to the cloud.

After the data collected at the edge is directly inferred by the edge device with built-in Intel CPU, the trained AI model can be deployed, optimized, and inference accelerated through OpenVINO™. For the data results obtained by inference at the edge, it can be uploaded to the cloud through logs or diagnostic results, and further processed in the cloud to generate larger-scale and in-depth analysis, or form a large-scale data dashboard, thus helping enterprises massively The visualization of data management is convenient for enterprises to manage various business levels.

In addition, after the edge data forming new training samples is uploaded to the cloud, it can also assist the existing model in the cloud to further train, iterate, optimize and upgrade, perform in-depth analysis, report processing and model retraining, and then download the retrained new model to the cloud. When the data reaches the edge, the edge can use the new model for more accurate reasoning.

Therefore, the edge-cloud collaboration method realized by OpenVINO™ can bring higher-quality AI solutions to enterprises to solve many pain points in practical use scenarios. At present, the edge-cloud integration solution realized by OpenVINO™ has two landing cases of smart medical care (AI cervical cancer pathology screening), smart manufacturing (AI real-time product defect detection), and the AI solution based on OpenVINO™ smart park.

How to implement real-time inference deployment of high-speed data collection at the edge through OpenVINO™

In many cases, GPUs with high power consumption and high cost are not necessary at the edge, so enterprises or developers will consider whether they can deploy and infer neural network models at the edge through the CPUs already used in existing equipment. The newly upgraded OpenVINO™ (version 2.2.1) is a solution that can help you achieve close to or achieve real-time AI model inference on the edge on the CPU.

In order to facilitate developers to use the open source tool suite, OpenVINO™ (version 2.2.1) has designed a simple "developer's journey". It only takes three steps of " create, optimize, deploy " to create a model for training or download open source pretrained model.

In the "Create" step, Open model zoom of OpenVINO™ (version 2.2.1) provides more than 270 pre-trained models (over 400 Baidu Feibao pre-selected models) verified and optimized by Intel, which is very convenient to use.

In the "optimization" step, after the developer obtains the AI model to be used, a set of optimization tools (MO, NNCF, POT) provided by OpenVINO™ (version 2.2.1) can help the developer to compress the model size and improve inference at the same time. speed.

In the "Deployment" step, it is convenient for you to deploy the model on each hardware platform in the actual use environment.

At present, OpenVINO™ (version 2.2.1) has supported the optimization, deployment and inference acceleration of models trained by multiple deep learning frameworks, such as the well-known deep learning frameworks such as TensorFlow, PyTorch, and Baidu Paddle.

Deployment Benefits of OpenVINO™ (2.2.1)

OpenVINO™ (version 2.2.1) can be deployed on multiple hardware platforms such as Intel CPU, IGPU, and VPU, and can run on different operating systems, such as Windows, Mac, and Linux. OpenVINO™ (version 2.2.1) also provides a very rich installation method.

1. Write code once and deploy it multiple times on multiple hardware platforms

By selecting different device names on the command line, you can directly deploy the model to different places by simply changing the device name. For example, when the device name selects CPU, the model can be deployed on a CPU similar to grey; when the device name is changed to GPU, the model can be run on an integrated GPU of the IGPU; when the device name is changed to varia, it can be Run it on movidias' VPU chip.

OpenVINO™ (version 2.2.1) also provides heron mode to support more deployments. After adopting the heron mode, the internal operations of the model suitable for running on the GPU can be run on the GPU, and the calculations suitable for the model running on the CPU can be run on the CPU.

In addition, there is a multi mode, which can run all the calculations of the model on the GPU and CPU in parallel, which can maximize the use of computing resources to run in parallel, and run inference faster.

Developers can do inference directly on CPU and GPU through OpenVINO™ (version 2.2.1) (GPU inference speed is relatively faster), but when doing AI inference on GPU, it often takes a lot of time to load and compile the model , so if developers use GPU inference, they often have to wait a long time to get the result of the first inference.

2. Auto-automatic device: CPU-accelerated inference

In the face of the above problems, the new version of OpenVINO™ (2.2.1) added the option to replace the device name with "auto automatic device", which perfectly combines the characteristics of CPU and GPU, and can solve the problem of high throughput and low throughput in different scenarios. Inference performance requirements such as delay allow developers to see inference results immediately without waiting for a long time.

In the past, developers needed to cross multiple hardware platforms such as CPU, GPU, and BPO when developing applications, and needed to write a lot of codes for different hardware to complete the switching of such complex hardware platforms. Now, just replace the device name with auto to deploy the model directly on the "auto" device, and the auto device called "performance hints" is equivalent to refactoring on the CPU and GPU plugins virtual plugin,

Other highlights: Azure+OpenVINO™, PaddlePaddle+OpenVINO™

The newly upgraded OpenVINO™ (version 2.2.1) adds support for natural language processing, speech and other fields and acceleration of optimized reasoning on the basis of previous vision, including dynamic support for natural original processing models, which can solve the problem. Questions about long and short sentences when the machine asks and answers.

At the same time, OpenVINO™ (version 2.2.1) also supports the deep learning framework of Baidu open source PaddlePaddle, which brings more value to developers by supporting the deep integration of PaddlePaddle. For example, Paddle's bert, OCR, etc. OpenVINO™ (version 2.2.1) developers can directly read deployment and reasoning without any intermediate format conversion.

OpenVINO™ (version 2.2.1) also provides a "Benchmark App" tool that allows developers to test network performance on their own computers.

Of course, there are also open source OpenVINO™ nobooks resources on github, which provide many code examples for developers to download, so as to understand how OpenVINO™ reasoning and deployment in specific tasks and scenarios.

In addition, OpenVINO™ can also run on cloud resources such as Microsoft's Azure cloud. When Microsoft, which focuses on cloud computing power, meets OpenVINO™, which focuses on hardware integration, the solution based on Microsoft Azure+OpenVINO™ has provided a landing scene for many enterprises (such as Advantech). Through Azure+OpenVINO™, the results of learning and inference are run on OpenVINO™, making AI development faster, more efficient and more convenient.

write at the end

Intel OpenVINO™ is a one-stop AI development tool launched by Intel in 2018 that can perform AI inference on CPU/GPU. Its applications extend from edge computing to enterprises and clients.

On the eve of this year's Mobile World Congress 2022 in Barcelona, Intel has launched a new version of the Intel distribution OpenVINO™ toolkit, adding new hardware auto-discovery and auto-optimization capabilities, as well as more deep learning model choices, more Device portability options along with higher inference performance and fewer code changes allow software developers to achieve optimal performance on any platform.

As a full-model tool chain, the newly upgraded OpenVINO™ can be regarded as the core of Intel's entire complete ecological tool station. It can be used for heterogeneous platform acceleration on different platforms on the deployment side, and it can improve performance and reasoning on CPU/GPU, effectively reducing development costs. Developer development threshold, empowering the continuous evolution of the entire AIoT ecosystem.