Author | OpenYurt Community
The emergence of cloud computing has promoted the explosive growth of the Internet of Things. Under the trend of increasing equipment scale and business complexity, edge computing has developed rapidly because of its ability to bring computing power closer to the edge of the network and equipment, thereby reducing the cost of cloud performance.
It is true that the edge computing of the Internet of Things is still in the early stages of development, and there are many challenges that need to be solved. For example, in a heterogeneous environment with a large number of software and extremely complex communication protocols, it is necessary to have the ability to quickly process business data and respond quickly to abnormal situations; in addition, in most cases, for security or other considerations, edge nodes Physically, it cannot be directly accessed from the cloud node, making deployment difficult, and it is impossible to achieve cloud-to-edge management. These problems threaten the continuity, stability and availability of the business.
Now, companies and developers can find solutions to the above problems through the open source community. Recently, OpenYurt and the open source project eKuiper formally reached a cooperation and completed the integration and docking: Starting from the v0.4.0 version, OpenYurt will officially support the deployment and management of eKuiper. The two parties will work together to help developers easily and efficiently solve the offline IoT edge computing scenario. Data processing and operation and maintenance challenges.
eKuiper: Lightweight IoT data analysis and stream processing open source software
Many scenarios of IoT edge computing require streaming data processing capabilities. The so-called streaming data refers to a set of sequential, large, fast, and continuous data sequences. In general, streaming data can be regarded as a dynamic data collection that continues to grow indefinitely over time. It can help users understand the status of system equipment in real time and respond quickly to abnormal situations.
On the edge side, computing resources (CPU, memory, etc.) are not as abundant as in the cloud, so traditional streaming data processing frameworks are similar to Apache Spark or Apache Flink, etc., because the installation package is too large, or the deployment structure and process are too complicated , High runtime consumption and other reasons, it is not suitable for running on these resource-constrained edge devices (industrial computers, gateways, or devices with low configuration such as X86 or ARM servers). and eKuiper is designed and developed to solve these problems on edge devices of the Internet of Things.
The predecessor of eKuiper was the Kuiper project officially open sourced in 2019 by the open source IoT data infrastructure software provider EMQ. In June 2021, the Kuiper project joined the LF Edge Foundation and changed its name to eKuiper, starting to operate as an independent project. The essence of eKuiper is a lightweight IoT data analysis and stream processing software that can run on various resource-constrained edge devices. It is hoped that the edge stream data processing has the capabilities of Spark and Flink.
As shown in the figure below, the overall architecture of eKuiper is roughly divided into three parts:
- On the left is Sources, which represents the location of the data source. The data source may be the MQTT Broker deployed at the edge of OpenYurt, or it may be message queues, files, and databases;
- On the right is Sinks, which represents the location to be stored after data processing is completed, that is, the target system. The target can be MQTT, which can be stored in a file or database, or HTTP service can be called;
- The middle part is the runtime of eKuiper, and the top layer is data business logic processing. This layer provides SQL and rule parsers, which are processed by the SQL processor and converted into SQL execution plans; the lower layer is flow runtime and SQL execution When running, run the execution plan that is finally executed; the bottom layer is storage, which stores some information that needs to be persisted during the running process.
In eKuiper, the user may dashboards to manage one or more instances eKuiper. Typically, these dashboards are deployed in cloud nodes to manage eKuiper instances across multiple edge nodes. As mentioned above, in most cases, edge nodes are physically inaccessible from cloud nodes, which makes deployment difficult and cannot perform efficient eKuiper cloud edge management.
OpenYurt changes this situation.
**
OpenYurt: a non-intrusive edge cloud native intelligence platform
Cloud native technology has become ubiquitous and has been applied to "new application loads", "new computing forms" and "new physical boundaries." As the cornerstone of cloud native technology, containers and Kubernetes are being carried by more and more computing forms, and the rich forms are beginning to move from the traditional central cloud to edge computing and to the terminal.
Generally speaking, in the edge scenario, the computing scale is huge and the business is complex. The workload management model of native Kubernetes is far from meeting the actual needs of cloud native edge computing; and the cloud edge network is connected through the public network, and the network connection is very uncontrollable. Factors, which may cause unstable factors in the operation of edge services, and because edge nodes are generally located inside the firewall of the user network, it will cause the objective condition that the cloud edge network can only be connected in one direction, which brings a lot to the native Kubernetes operation and maintenance monitoring Challenge: Finally, it is inevitable to face the diversified and heterogeneous types of edge resources, making it difficult to support edge standardization.
OpenYurt is built on native Kubernetes and is the industry's first cloud-native open source platform for edge computing that is non-invasive to Kubernetes. OpenYurt is the core framework of Alibaba Cloud's container service product ACK@Edge. In May 2020, Alibaba Cloud will feed back the industry in an open source manner, and through donations to CNCF, it will achieve a more neutral and open community environment and become the lower edge of ecologically compatible scenarios. The preferred project of cloud native platform.
As shown in the figure below, OpenYurt's architectural design is very simple and is a typical "center-edge" model. By adding Yurt Controller Manager, Yurt App Manager and Tunnel Server components on the cloud (K8s Master). On the edge (K8s Worker), YurtHub and Tunnel Agent components are added:
This architecture design gives the following enhancements to the native Kubernetes capabilities in the edge scenario:
- Edge : Through the Yurt App Manager component, manage edge resources scattered in different regions from the perspective of unitization, and provide independent lifecycle management, upgrades, expansion and contraction, and traffic closed loops for businesses in each region. And other capabilities; and the business does not require any adaptation or transformation
- Edge Autonomy : Because each edge node adds a transparent proxy YurtHub with caching capabilities, it can ensure that the cloud edge network is disconnected. If the node or business restarts, the local cache data can be used to restore the business
- Cloud Edge Collaboration (Operation and Maintenance Monitoring) : Through the cooperation of Tunnel Server/Tunnel Agent, it provides a secure cloud edge two-way authentication encrypted channel for edge nodes located inside the firewall, even in edge computing scenarios where the edge-to-cloud network is one-way connected Users can still run native kubernetes operation and maintenance commands (such as kubectl proxy/logs/exec/port-forward/attach, etc.). At the same time, centralized operation and maintenance monitoring systems (such as prometheus, metrics-server, etc.) can also obtain edge monitoring data through cloud-side channels
- Cloud native ecosystem compatible with : All functions are enhanced through Add-on or controller form to enhance Kubernetes, so 100% compatibility with Kubernetes and cloud native community ecology is guaranteed; In addition, OpenYurt also provides a YurtCtl tool that can be used for native One-click conversion between Kubernetes and OpenYurt clusters
In short, OpenYurt enables users to manage applications running in edge infrastructure as if they were running in cloud infrastructure.
Use OpenYurt to deploy and manage eKuiper
Starting from the v0.4.0 version, the OpenYurt project officially supports the deployment of eKuiper and its dashboard. Users can use the yurt-tunnel cloud-side tunnel to implement configuration management of eKuiper instances through eKuiper’s web management console, realizing efficient and lightweight IoT Edge stream processing capabilities.
The eKuiper and eKuiper Dashboard components are directly deployed using native Helm Chart. The eKuiper component runs on the edge side, while the ekuiper Dashboard runs on the cloud (control side). At the same time, OpenYurt's Yurt-Tunnel component automatically intercepts cloud-side network communications through mechanisms such as dns and iptables dnat. Therefore, the eKuiper component can complete cloud-to-edge instructions without being aware of cloud-side tunnels.
Based on the above methods, users can directly use the dashboard on the OpenYurt platform to easily manage eKuiper in edge scenarios, and use the dashboard in the browser to check whether the service is healthy in time:
Community cooperation planning
In the future, both parties will continue to explore efficient cloud-native edge middleware solutions for the IoT industry, and attract more lightweight IoT-related middleware to join the OpenYurt ecosystem, so that edge middleware can truly be deployed in a cloud-native way. In addition, as a contributor to eKuiper, EMQ will officially become a member of the cloud-native IoT SIG initiated by OpenYurt after Alibaba Cloud, VMWare, and Intel.
I hope that more developers will support and join the OpenYurt and eKuiper communities to build a cloud-native IoT open source ecosystem.
Scan the QR code and join the OpenYurt open source community:
Scan the QR code on WeChat to join the eKuiper open source community:
Extended reading
If you want to know more features of LF Edge eKuiper and CNCF OpenYurt, please read the following references:
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。