NetEase Shufan Envoy Gateway Practical Journey: Sticking to 6 Years

The maturity of service meshes continues to improve, and traffic processing in the cloud-native environment becomes more and more important. The Envoy Gateway project was recently announced as open source, "aiming to greatly reduce the threshold for using Envoy as an API gateway", which has attracted industry attention. In November 2018, Envoy became a CNCF graduation project and began to be known to practitioners as a high-performance data and service proxy software. However, in the following two or three years, in domestic API gateway practice, the selection of Envoy is still a relatively lonely road. .

As a pioneer of domestic cloud native practice, NetEase Shufan Qingzhou cloud native team explored the realization of service mesh platform and API gateway based on Istio and Envoy as early as 2017, and completed the mature application of this system in Yanxuan e-commerce business in 2019 . After 6 years of practice, on the enterprise side, NetEase Shufan Envoy Gateway carries the core business traffic of leading enterprises in the Internet, banking, securities, energy and other industries, and has withstood the test of tens of billions of daily traffic; on the community side, the team The Hango Gateway based on Istio and Envoy will be open sourced in 2021, and the only Envoy Maintainer in China will be born in 2022.

In the process, NetEase Shufan has also shared the relevant practices of Envoy Gateway many times, and discussed the technical route of the gateway in 2020 to demonstrate the advantages of Envoy Gateway, and jointly promote and witness the continuous maturity of Envoy with the community. NetEase Shufan believes that the open source of Envoy Gateway has brought a good start for Envoy to become a standard data plane component.

NetEase Shufan Cloud Native Architecture Selects Envoy

In the construction of the cloud native technology system, NetEase Shufan attaches great importance to the unified technology stack, and believes that only in this way can the R&D cost be reduced and the advantages of cloud native be truly brought into play. The grid gateway system chooses Envoy not only because Envoy is the default sidecar on the Istio data plane, but also because it is a "double eugenics" with excellent functions and performance. Since then, the practice of strict selection of business in NetEase has verified the correctness of this choice.

On the grid side, NetEase Shufan believes that Istio + Envoy's good abstraction of microservice traffic and service governance has brought the dawn of a unified service layer technology stack. At the same time, Envoy has a forwarding performance not lower than that of Nginx, but it is much more flexible than Nginx in terms of governance and control capabilities (UDPA). In Netease Yanxuan's test at the time, eBPF/xDP (sockops) was used, and the optimized path was SVC <-> Envoy, and the latency performance was improved by 10-20%.


The upgrade of NetEase Yanxuan Gateway considers the seamless integration of the Qingzhou micro-service system and the implementation of mainstream products. It also uses the Envoy data plane component, which is responsible for the proxy, routing, governance, telemetry, etc. of the north-south data traffic; through filterchain to expand, Supports the writing of plug-ins based on Lua and C++ languages, and supports multi-language expansion after WASM is implemented; and performs dynamic control such as configuration and distribution through xDS and control plane components. The control plane uses Istio Pilot as the basic control plane component, and provides an API layer and console for users or third-party platforms to access.

Based on Qingzhou Envoy Gateway NetEase Yanxuan realized:

  1. The gateway management platform is reused to ensure the consistency of user habits.
  2. LUA plug-ins are reused to facilitate seamless migration of extended functions.
  3. The support of function-level routing capabilities paves the way for subsequent FaaS drainage.

After the implementation of large-scale business production, NetEase Shufan has realized the innate advantages of Envoy, and firmly believes that Envoy Gateway is the standard technical solution for cloud-native business traffic entry:

  1. Richer features than HAProxy and Nginx
  2. Comparable to Nginx, much higher than the performance of traditional API gateways
  3. Strong dynamic management and control capabilities, with the data plane standard xDS protocol
  4. Natural Affinity Container Environment
  5. Multilingual extension sandbox - WASM

In terms of performance, in the test of Netease Shufan, the TPS of Envoy can reach about 12W, while the TPS of Kong based on Nginx is about 5W.

By 2020, Qingzhou Envoy Gateway will be implemented on a large scale in multiple core businesses of NetEase:

  • NetEase Media (News) has realized that the entire site traffic is exposed through the Qingzhou Envoy Gateway
  • NetEase Yanxuan has realized that all traffic of cloud services is exposed through Qingzhou Envoy Gateway
  • NetEase Youdao, Yunxin, Lofter and other NetEase core Internet business traffic are exposed through Qingzhou Envoy Gateway


Hango Open Source, Enter CNCF Landscape

In August 2021, NetEase Shufan open sourced Hango, a high-performance, scalable, and feature-rich cloud-native API gateway, and then comprehensively interprets Envoy technology in terms of functionality, performance, industry impact, technology trends, and best practices. The advantages of the route, as well as the extended design and landing practice of Hango.

In short, the data plane of Hango is extended based on Envoy to enhance the plug-in chain, and the control plane is extended based on Istio, and supports multi-scenario capabilities such as microservice gateway, seven-layer load balancing, and Kubernetes Ingress. The following is the data flow of the Hango gateway plug-in chain. By creating an EnvoyPlugin CR, Slime dynamically monitors and aggregates to generate the corresponding EnvoyFilter to complete the dynamic expansion of the Envoy filter chain.

And Hango has also been recognized by cloud native practitioners and entered CNCF Landscape.


Looking forward to the future, NetEase Shufan is committed to expanding the "out-of-the-circle" scale of the Qingzhou cloud native system and integrating it into industrial digitization. As a core module, the Qingzhou Envoy Gateway will also increase the landing capabilities of industrial application scenarios , such as multi-cluster high availability, protocol conversion and other financial scenarios Enhancement of just-needed capabilities. At the same time, more capabilities of Qingzhou Envoy Gateway will be open sourced through Hango Gateway.

Continue to contribute, the only Maintainer in China is released

At present, the NetEase Shufan Qingzhou team has contributed 60+ PRs and more than 14,000+ new codes to the Envoy community, covering core functions such as Envoy's stateful session retention, enhanced tracing capabilities, Lua script support, and enhanced Dubbo governance capabilities.

In March 2022, the Envoy community invited Wang Baiping, a cloud native expert and senior architect of NetEase Shufan, to become the community Maintainer - this is the first and only Envoy Maintainer in China, and is also a Dubbo Extension Senior Maintainer, indicating that the community continues to contribute to NetEase Shufan 's approval.


Envoy Maintainer Helps Unlock New Envoy Skills

Before becoming an Envoy Maintainer, Wang Baiping also actively interpreted the development of Envoy technology and related practices including Envoy Gateway through articles, live broadcasts, and offline sharing.


Finally, readers and friends are welcome to actively participate in the Envoy and Hango communities to create a cloud-native future.

From May 13th to June 15th, 2022, the Loggie community launched the Loggie Geek Camp open source collaboration event for cloud native, observability and log technology enthusiasts. Feel the essence of open source culture and the creativity of the open source community to create the future of cloud-native observability. It includes four types of tasks, including providing user cases, catching bugs, improving and submitting features. The submitted content is considered a success if it passes the community review. Those who perform well will be commended by NetEase Shufan and the Loggie community . Welcome to visit the link to learn and participate: https://sf.163.com/loggie



391 声望
545 粉丝
0 条评论
针对业务场景中数据应用价值的落地,网易数帆形成了以 DataOps、DataFusion、DataProduct 为内核,数据技术、数据资产、数据应用和数据运营为四要素的数据生产力模型,其中网易公司数据运营的一个重要手段是网易...

网易数帆阅读 170


spacewander3阅读 1.8k评论 2

深入剖析容器网络和 iptables
Docker 能为我们提供很强大和灵活的网络能力,很大程度上要归功于与 iptables 的结合。在使用时,你可能没有太关注到 iptables 的作用,这是因为 Docker 已经帮我们自动完成了相关的配置。

张晋涛3阅读 1.2k

Kubernetes v1.26 新特性一览
我每期的 「k8s生态周报」都有一个叫上游进展的部分,所以很多值得关注的内容在之前的文章中已经发过了。这篇中我会再额外介绍一些之前未涵盖的,和之前介绍过的值得关注的内容。

张晋涛2阅读 573评论 1

Kubernetes 证书管理系列(一)
大家好,我是张晋涛。这是一个系列文章,将会通过七篇内容和大家一起聊聊 Kubernetes 中的证书管理。以下是内容概览:如上所示,在第一篇中,我们将从原理出发,来理解 Kubernetes 中的证书及其相关的作用,然后...

张晋涛2阅读 803

CodeGalaxy 推出轻量集群,可在云主机上一键搭建 K8s
CodeGalaxy 是 Swoole 官方推出的 ServerLess 平台,底层基于 Docker 和 K8s,帮助开发者更简单方便地管理云上的 Web 应用/服务。CodeGalaxy 是完全免费的,用户不需要付费即可使用。

韩天峰2阅读 400

企业流程数字化转型研讨会暨《流程优化风暴》新书发布会 即将召开

MissD阅读 4.8k


391 声望
545 粉丝