The case of using Microsoft cloud AIoT technology to accelerate the implementation of the project - illegal parking detection

AIoT is a combination of AI (artificial intelligence) and IoT (Internet of Things) technologies. Graphics and image processing is an important branch in the field of artificial intelligence. In daily life, there are also a large number of scenarios based on graphic and image processing: such as traffic violation snapshots , vision-based anti-fatigue monitoring for drivers, home cameras for elderly people to fall alarms and other functions, using the Internet of Things and edge computing technology, the cloud-trained model can be pushed to the edge more easily, thereby achieving more efficient reasoning at the edge Tasks, the integrated application of AIoT also has huge market prospects in scenarios such as smart homes, connected cars, and smart supply chains. With the development of science and technology, more and more sub-scenarios require the combination of AI technology and IoT technology based on graphics and images as much as possible. Detection, vision-based ADAS advanced driver assistance systems, etc. In the face of these growing demands, how to quickly implement an AIOT project has become more important. Several very interesting tools can help us quickly implement AIOT innovation.

Researcher at Microsoft MVP Labs

Next, let's take "fire exit, no occupation" as an example to see how to use the tools provided by Microsoft Cloud to implement this scenario.

Azure IoT Hub

Azure IoT Hub is a cloud "gateway" service that can host "millions of" devices. It mainly provides IoT device authentication, two-way communication between devices and the cloud, device OAT, device management, security and other functions.
Let's take a look at the scenarios corresponding to these functions in turn:

• Device Authentication:
  Device authentication is the first step in the interaction between the device and the cloud. The legal identity of the device must be verified before the device is allowed to initiate a data request to the cloud.
• Two-way communication between device and cloud:
  The so-called two-way communication, such as smart air conditioners, reports the room temperature from the air conditioner to the cloud and then displays it on the mobile phone. The process of "reverse control" the device
• Device OTA:
  Device OAT (Over-the-Air Technology), the loudest scene of this technology should be the smart car to upgrade the latest functions through OTA, in fact, OTA usually includes the update of the device operating system level and the update of the application on the operating system
• Device management:
  This is an important foundation of the IoT system. The so-called device management is mainly to maintain the "database" of the devices contained in an IoT system. This database records how many devices can be connected to the platform, and the current status of each device. What is the running state, whether these devices are logically grouped, such as certain escalators in a shopping mall, which devices can perform some time-consuming operations at what time period, such as a car in Shanghai with hardware version 1.0.1 It can be automatically upgraded after confirmation by the user at 2 am on Sunday, etc.
• security function:
  Security function is also an important function. Counting the history of IOT security incidents, from smart fish tanks to smart cameras, smart doorbells have all exposed security vulnerabilities. Some vulnerabilities may leak user privacy, and some may bring life danger, such as Once medical equipment is illegally attacked, the consequences are disastrous. IoT security will also be considered from several levels: 1. Operating system security, 2. Application security on the system, 3. Communication security, etc.

Azure IoT Edge

Azure IoT Edge is a collection of applications running on "edge computing devices", primarily a runtime that includes the Moby container engine and a set of built-in containers. This set of applications can be installed on most Linux or Windows devices, ranging from Widows-based subway turnstiles, bank ATMs, to Linux-based Raspberry Pis, drones, and other devices that can run Azure IoT Edge , At the same time, since it is a container-based runtime, theoretically, any service that can be containerized can also run on this platform, such as Mysql, Nginx, etc. Of course, some cloud services from Microsoft's original factory can also run in the form of containers On edge devices like Azure Functions, Azure SQL Edge, etc.

The main reasons for deploying applications on edge computing devices are as follows:

• Integrated ability to interact with Azure IoT Hub:

The device management, two-way communication, OTA and other capabilities of Azure IoT Edge are connected with Azure IoT Hub. It is a solution that extends from the Hub in the cloud to the edge.

• There are some built-in scene functions:

Azure IoT Edge has built-in support for some industrial protocols, such as Modbus, OPC-UA, etc., for industrial and other scenarios.

• Lower latency and stronger computing power sinking:

In scenarios with massive video processing, latency is a very important indicator. For example, in a 5G smart port that the author has implemented, the tower crane needs to have some video-based analysis to take pictures and analyze the lifted container marks and record them. In the container assembly area, it is assumed that once the results are returned 20 seconds after the photo is taken, the overall loading and unloading efficiency will be greatly reduced. Therefore, a special video processing server with GPU is usually placed near the tower crane, and the tower crane camera is constructed using a 5G private network. And the GPU server is deployed in a "local area network" supported by 5G, the latency is greatly reduced.

• Can deal with network strobe:

In some underground parking garages or highways, the 4/5G signal may be unstable. At this time, edge computing equipment can also be used, and then the parking service or high-speed video capture service can be placed "locally", even if the network is not available. Well, the communication between the terminal and the "edge computing device" can still be completed.

• Addressing Data Sovereignty Issues:

In some factories, production data needs to be stored locally, and at the same time, alarm data is expected to be sent to the cloud. In this scenario, edge computing devices can also be deployed inside the factory, and SQL Edge can be deployed on the edge devices to record local data, and in Deploy Azure Functions or Stream Analysis Service locally to filter and analyze local data and send alarm data.

• Cloud Edge Expansion Capability:

The cloud edge extension mainly extends some capabilities of the cloud to the edge. There are many original services on Azure, which can be deployed in the form of containers. Most of these services can also be deployed on local edge devices, such as custom vision, SQL Edge, etc. .

• The ability to deploy containers remotely:

By using Azure IoT Edge and Azure IoT Hub together, a container deployment manifest can be configured in the cloud, and IoT Edge will automatically pull containers from the cloud and deploy them on edge computing devices according to the contents of the manifest.

After introducing the above content, we summarize the main functions of IoT Edge in the form of a picture:

Azure Custom Vision

Azure Custom Vision is a service that can implement image classification and image recognition. If we want to make an application for classifying tigers and elephants, we only need to provide dozens of tiger photos and dozens of elephant photos, and then Annotate on Custom Vision, and after training through the Custom Vision service, you can generate an online API or export it as a container or other algorithm model, and realize object classification through simple HTTP API interaction.
Possible training set scenarios to consider when using Custom Vision to train a fire escape occupancy algorithm:

• Vehicles of various colors
• various types of vehicles;
• head direction;
• Weather factors (including night and day, etc.)

After using Custom Vision to train the fire escape occupancy model, the container can be exported and an HTTP inference endpoint can be provided. The exported container can be deployed to remote edge devices through IoT Hub.
Custom Vision modules deployed on edge devices also already have the ability to classify images.
Next, let's look at how to implement real-time video analytics on edge devices using Azure or third-party tools and services.

Real-time video analytics on edge devices

The real-time video analysis on the edge device mainly includes video decoding, processing the video into frames and sending the video frames to the AI inference server. The AI inference server returns the results and performs message processing or video processing according to the returned results. When we deploy the video decoding and AI inference service on the same computing node, we can send video data to the inference service through shared memory to perform inference at a rate of 30 frames per second.

▌Azure video analyzer preview (for reference only, will be retired soon)

Azure Video Analyzer (preview) provides a platform for building intelligent video applications across the edge and the cloud. The platform consists of an IoT Edge module and an Azure service. It provides the ability to capture, record and analyze live video and publish the results (that is, the video and the insights from the video) to the edge or cloud. AVA has built-in support for RTSP and can directly connect to most video recorders.

AVA integrates pipeline capabilities for real-time video analysis using gRPC or HTTP: supports two gRPC and HTTP extensions to integrate video analysis algorithms:

An extension node in AVA acts as a proxy that samples incoming video frames and converts them to a specified image type. It then forwards the image via REST/gRPC to another Edge module that runs an AI inference model behind the endpoint, which can be a custom developed container or a container exported by custom vision. Extension processor nodes collect detections and publish events to IoT Hub message sink nodes. The node then sends these events to the IoT Edge Hub, and messages arriving at the IoT Edge Hub can be stored locally or filtered and uploaded to the cloud IoT Hub.

▌Intel OpenVINO™ DL Streamer integration

Although AVA is about to be discontinued, we can still apply the AVA idea to integration with other services, such as Intel OpenVINO™ DL Streamer Edge AI also provides similar functionality, the OpenVINO™ DL Streamer Edge AI extension module is an Intel-based Microservice for Video Analytics Service (VA Service) that serves video analytics pipelines built with OpenVINO™ DL Streamer. Developers can send decoded video frames to an AI extension, which performs detection, classification, or tracking and returns the results. The AI extension exposes gRPC APIs that are compatible with video analytics platforms such as Microsoft's Azure Video Analyzer.

The usage logic of OpenVINO™ DL Streamer is basically the same as that of AVA:

Insights and Actions - Integration of the Power Platform Family

Real-time video analysis scenarios on edge devices can be integrated with the Power Platform family. For example, when a fire exit is occupied, a message can be pushed through Power Automate and Power Apps:

Inference events can be further visualized using Power BI and Azure Stream Analytics:

Use the player component in Power BI to watch the corresponding alarm video synchronously:

Microsoft Most Valuable Professional (MVP)

The Microsoft Most Valuable Professional is a global award given to third-party technology professionals by Microsoft Corporation. For 29 years, technology community leaders around the world have received this award for sharing their expertise and experience in technology communities both online and offline.
MVPs are a carefully selected team of experts who represent the most skilled and intelligent minds, passionate and helpful experts who are deeply invested in the community. MVP is committed to helping others and maximizing the use of Microsoft technologies by Microsoft technical community users by speaking, forum Q&A, creating websites, writing blogs, sharing videos, open source projects, organizing conferences, etc.
For more details, please visit the official website: https://mvp.microsoft.com/zh-cn

Go to the official Azure IoT Hub documentation for concepts and how-to guides, and how to integrate IoT Hub with other Azure services to generate a complete end-to-end solution.

Click to go to the official documentation of Azure IoT Hub~