NetEase X ICBC: Cloud native log system Loggie is officially open source!

Recently, the cloud native log system Loggie project (github.com/loggie-io/loggie/) , jointly initiated by NetEase and the Industrial and Commercial Bank of China, was officially open . This is NetEase Shufan's attack on the pain point of cloud native logs, and it is also another action of the team and partners to implement the technical concept of "open architecture, open source kernel" and hand over control to customers.

Loggie Project: Cracking the Pain of Cloud Native Logs

In the wave of enterprise digital transformation, the use of cloud native technology to solve new challenges in digital software R&D and O&M has become a mainstream choice. However, in the cloud native environment, large-scale and frequent dynamic migration of containers, diversity of log storage, and Kubernetes metadata query, etc. Feature , forcing a change in the way logs are managed. With the deepening of business practice, problems such as more and more human flesh operation and maintenance in logging, difficulty in expansion of function development, and difficulty in supporting larger scale gradually surfaced. The existing open source solutions in the industry failed to meet the needs, such as:

• The performance is not up to standard, or the development efficiency of extended functions is low, and a good balance is not achieved

• Limited support for containerized scenarios

• Most open source projects do not provide a complete logging solution

In this context, Loggie for cloud-native scenarios came into being. Loggie is a lightweight, high-performance, cloud-native log collection Agent and transfer processing Aggregator based on Golang, which supports multiple pipelines and hot-plugging of components, and provides:

• One-stack log solution: supports log transfer, filtering, parsing, segmentation, log alarming, etc.

• Cloud-native log form: fast and convenient container log collection, native Kubernetes CRD dynamic configuration and delivery

• Production-level features: Based on long-term large-scale operation and maintenance experience, it has formed a full range of observability, rapid troubleshooting, abnormal early warning, and automated operation and maintenance capabilities

Combined with the long-term business practice of NetEase Shufan, Loggie has formed the following major characteristics:

• Extensible and hot-swappable: is configured with different Source/Interceptor/Sink, and has the capabilities of transfer, filtering, parsing, segmentation, log alarm, etc., and can use Golang to quickly develop self-developed plug-ins

• Strong isolation: multi-pipeline design reduces mutual interference and can send multiple different data sources at the same time

• Lightweight and high performance: based on Golang, with very little resource usage and strong throughput performance

• Reliability guarantee: complete log observability, native Prometheus metrics support, and Interceptors such as limited flow

• Cloud native : The configuration center integrates Kubernetes, and you can collect container logs by creating a CRD instance

At the same time, based on Loggie's streaming data transmission design, we can use Loggie's Pipeline flexibly.

From the form of use, it can be divided into:

• Agent form: per node or one per Pod, used to collect logs or other data

• Aggregator form: is used for transit, forwarding and processing, and can be independently deployed as a cluster

Can be applied to:

• Data collection: collects container logs, node logs, Prometheus metrics, Kubernetes Events, etc.

• Data transfer: is used as a transfer machine for data aggregation, forwarding and distribution

• Data processing: performs streaming data segmentation, conversion and processing

• Log alarm: performs abnormal log detection and alarm

 ……

In addition, using Loggie's design, users can also quickly develop a Source, Sink or Interceptor component, reuse Loggie's capabilities, and avoid a lot of repetitive development work, such as:

• In the Kubernetes cluster, CRD can be used to issue configuration conveniently and directly, and it supports automatic reload and specified Loggie cluster, without considering deployment, configuration update and other issues

• Rely on Loggie to provide the stability and reliability of the transmission process, ensure at-least-once and retry mechanisms, avoid data loss, and the hidden dangers caused by excessive or excessive data volume

• Using a series of monitoring indicators provided by Loggie, such as queue length, transmission delay, sending QPS, etc., you can quickly access Prometheus, and you can also use some built-in interfaces and capabilities for quick troubleshooting

• Use pluggable Interceptor for custom data processing, format conversion, etc., avoiding excessive custom code development

Loggie Ecosystem: Joining Hands with Partners to Strengthen System Capabilities

Loggie sprouts from the actual needs of NetEase Yanxuan business, grows from the long-term joint construction of Yanxuan and Shufan, and continues to develop in the close cooperation between NetEase Shufan, NetEase News and Industrial and Commercial Bank of China. 's extensive ecology enables the project to continuously improve and mature based on business needs.

Before Loggie, NetEase Yanxuan used an open source log solution. Since there are multiple log collection scenarios such as virtual machines, physical machines, and containers in the business, the needs to cope with the large-scale traffic that is equivalent to 10 times more than the peak period. Operation and maintenance and other aspects have encountered very big pain points. Therefore, in 2019, Yanxuan and Shufan started to build a new cloud-native logging system, Loggie.

, General Manager of NetEase Qingzhou Products, said:

Loggie is an important achievement of our cloud-native practice. The project follows the cloud-native community standards, is compatible with traditional technology architecture, and integrates the experience of different industry application scenarios such as ICBC. Enterprises can use Loggie to unify log management and smooth cloud-native applications. Landing escort. The maturity of Loggie is due to the joint construction of NetEase Shufan and the business. We believe that open source will enable the project to adapt to more industry scenarios, enrich the open source technology ecosystem, and promote the development of cloud native technology.

---

, a senior technical expert at NetEase and the head of , said:

Loggie has been fully verified in the mixed production environment of NetEase Yanxuan for more than 2 years. Now, with the strong support of the group, we have open sourced Loggie. On the one hand, we hope to fill the gap in the community's one-stop log solution and help more enterprises that encounter similar problems; on the other hand, we also hope to With the power of the community, the Loggie project will continue to grow and achieve a more sustainable and healthy development.

In the future, Yanxuan will continue to participate in the development and management of the Loggie project based on its own business scenarios and technical accumulation in cloud native practice, and looks forward to more like-minded partners to participate.

---

Industrial and Commercial Bank of China has built the largest enterprise-level cloud platform in the industry, with 100% of core applications in the cloud, over 100 container clusters, and over 300,000 running containers. ICBC started to build an enterprise-level log center in 2017. The log center realizes the collection, cleaning, formatting and storage of various application logs, and supports user retrieval and download, with an average daily log capacity of 45TB; based on standardized log data to create a lightweight High-level operation services**, quickly and effectively support real-time acquisition of basic operation indicators such as business transaction rate and time consumption, and also provide convenient user views such as mobile terminals and front desks.

---

of Cloud Computing Laboratory of ICBC Software Development Center, said:

In terms of community contribution, ICBC, as a participating unit of the Loggie project, combined with the log platform construction experience, and carried out in-depth customized development in core modules such as collection target discovery, log interception processing, and component self-monitoring. In the future, ICBC will continue to participate in the work of the Loggie community, and plans to continue in-depth research on modules such as log output expansion and stream processing to feed back to the open source community.

The Future of Loggie: Meeting More Needs for

Loggie has been widely used within NetEase, Industrial and Commercial Bank of China, and NetEase Qingzhou's numerous commercial customer environments. In the actual measurement of NetEase's strict selection business, the CPU consumption of Loggie is only 1/4 of the previous solution, and the transmission throughput is 1.6~2.6 times of the latter's , and it even reaches 200MB/s+ under multi-pipeline collection Limit throughput .

After the open source release of Loggie, NetEase Shufan's open source self-developed cloud native projects have reached 6 , including cloud native software-defined storage system Curve, multi-tenant visual Kubernetes management platform KubeCube, cloud native operation and maintenance diagnosis system KubeDiag, high-performance Cloud native API gateway Hango, smart grid manager Slime, etc. Chen Chen previously said: NetEase Shufan's series of open source work in the cloud native field aims to facilitate the introduction of cloud native technology stacks for customers, and enhance the open source of basic components at the enterprise level to help customers solve cloud native infrastructure management, control, maintenance and stability issues.

The open source of Loggie means that NetEase Shufan has also fallen heavily in the field of observability of the cloud native open source layout, forming a more complete layout.

In the future, Loggie will provide more components and function extensions, launch lightweight streaming capabilities, continue to solve more pain points of cloud native logs, and meet more needs in the direction of . Friends who are interested in this field are welcome to learn about the Loggie project and participate in the construction of the Loggie community together!

understand more
Loggie project address: https://github.com/loggie-io/loggie/

Loggie project documentation: https://loggie-io.github.io/docs/

Overview of Shufan's independent open source: https://sf.163.com/opensource