Wang Yubo: Looking at the integration of cloud, open source and community from Amazon's practice | DEV. Together 2021 China Developer Ecological Summit

content source: On June 5, 2021, the 2021 China Developer Ecological Summit hosted by SegmentFault came to a successful conclusion. At the meeting, Amazon Web Services chief evangelist Wang Yubo delivered a keynote speech on "Viewing the Integration of Cloud, Open Source and Community from Amazon's Practice".

sharing guest: , chief evangelist of Amazon Web Services.

shorthand compilation and release: SegmentFault Editorial Department

Hello everyone, today my sharing is about some of Amazon's contributions and practices in the open source field.

I believe that everyone will have some understanding and consideration of open source. So today I would like to introduce to you from the perspective of cloud computing vendors, how we enable more developers to use open source software faster, more convenient and safer on the cloud, and help them carry out some applications and deploy.

In fact, whether it is for our customers, partners, or developers, many of you are concerned about can apply open source software faster and better . First of all, only by applying our open source tools well, will we consider how to make further contributions to open source and how to make some contributions to the community in subsequent practice.

Who is concerned about open source?

As a cloud vendor such as Amazon Cloud Technology, it has actually done a lot of work behind the scenes. For developers, there are many factors that pay attention to open source, such as personal hobbies, personal brand promotion, and career development. In fact, open source has a lot of attraction for different developers, and it also encourages many developers to make a lot of contributions around open source and the community. We can see that many developers have gained a lot of influence through open source in their careers, from which they, including his company and his community, have obtained continuous, strong and vigorous growth.

For enterprises, there are also many enterprises that are doing further open source. I have a set of data made by Black Duck in previous years. Currently in the enterprise software environment, 96% of the software contains open source-related codes or projects. Open source has increasingly become a very important part of enterprise core software architecture. Moreover, the use of open source can also enable our company to carry out rapid development, rapid practice, rapid iteration, and quickly bring our products to the market for our customers to make better use and better help, so that our company's innovation can Further improve.

We can see from the perspective of developers and companies that companies and developers are constantly using and embracing open source. For cloud computing vendors, we can see that many cloud vendors, including Amazon Cloud Technology, are Embrace open source actively. In fact, cloud and open source, the integration of cloud and community is also a very big trend. Here I want to quote a set of data. Among the top five open source vendors in the world, four of them are cloud computing vendors. Seven of the top ten households account for as many as seven. It can be seen that with the continuous development of cloud computing and open source, more and more cloud computing vendors are also embracing open source so that we can enable our developers to use services faster, more convenient and safer on the cloud.

For our customers and our developers, how do they view the open source leadership of cloud vendors? Here is a set of surveys we did:

In fact, this group of surveys is not only for customers of Amazon Cloud Technology, but also for many customers who do not use Amazon Cloud Technology services. According to our survey results, more customers are the same as we talked about before, and they also hope that open source software can be deployed quickly and conveniently in the cloud computing environment, so that open source software can quickly bring practicality to enterprises and individuals to improve production efficiency. the result of. Of course, many of our developers also hope that cloud vendors can help them provide a series of better ways to use open source, and also hope that they can make more contributions to open source through cloud computing.

In fact, from the perspective of developers and cloud vendors, their goals are generally the same, and they all hope that our open source software can be used quickly and conveniently in the cloud environment.

For Amazon Cloud Technology, the world's leading cloud computing vendor, in fact, it has been paying attention to the open source field. Of course, our voice in the open source field, compared to some other vendors, does not seem to be a very strong voice. But we have made a lot of efforts behind the scenes to help our customers, to help developers use and deploy open source software on the cloud.

Amazon has a very important leadership principle, called customer first , we have done a lot of innovation on behalf of our customers. We have easily deployed some open source software that is very difficult to extend in the production environment on the cloud. This allows our developers to quickly and easily build an open source environment that can be expanded quickly in the cloud through simple cloud services. At the same time, Amazon Cloud Technology has also initiated a series of open source projects to help developers solve the actual problems they encounter in the production process.

At the same time, we have also built a whole set of community system, through us, community evangelists, and some of our friends called'hero' to promote the further integration of our cloud services and open source, while also further expanding the community. Many employees in our company are participating in open source. We encourage employees to contribute and participate in related open source software. Of course, some are project-related, and some are personal hobbies of employees. At Amazon, the number of engineers contributing to open source is increasing year by year. The number of open source warehouses we have contributed to currently has as many as 2500 on Github, covering areas ranging from basic computing, containers, serverless to data, artificial intelligence, machine learning, robotics, etc.

We are making continuous contributions to the open source community in many areas. So here are some examples of open source projects that Amazon participates in. This is just a small part of our participation in the project.

We have contributed and participated in many open source projects, and we also support a series of open source technology projects through different channels. For example, through a series of financial support and cloud service use support to help our developers, help the community, quickly build a variety of participating applications on the cloud, and we also through a series of recruitment, development projects Ways to increase our support for developers and builders, enabling the entire Amazon Cloud Technology to build an open source system for developers.

Amazon Cloud Technology's contribution to the field of open source is mainly concentrated in two aspects. First, we hope to provide a series of cloud services through Amazon Cloud Technology to help developers use open source software on the cloud quickly, safely and conveniently. In the second aspect, we have also initiated a series of numerous new open source projects to solve the actual problems of our customers. This picture illustrates our series of efforts to quickly implement open source construction on the cloud for the first field.

Everyone knows that Amazon Cloud Technology currently provides 200 full-featured services, many of which are related to open source. We help open source software to be installed, used, and deployed on the cloud, which allows our developers to carry out convenient development and practice on the cloud. For example, on the cloud, we can easily use software like Cassandra, FreeRTOS, Kubernetes, etc., and at the same time we add a lot of observability, manageability and security monitoring measures to it, which makes us in a production-oriented environment Open source can be used for convenient and quick production and deployment. In addition, we have also packaged and released many brand new open source projects in the form of cloud services, which enables more customers to quickly experience the advantages of related open source in the cloud. For example, FireCraker-based Lambda is our leading serverless computing capability. Through cloud services, the latest open source capabilities can be better reflected in cloud services.

What open source projects have we participated in and contributed to?

course Rust, which is a very popular programming language. Actually, we started sponsoring the Rust open source project as early as 2019. At present, many of the main Rust contributors are working at Amazon Cloud Technology, including Felix Klock, one of the main leaders who recruited the Rust team compiler last year. He also joined Amazon Cloud Technology. At present, many of our cloud services, including Amazon S3, Amazon EC2, Amazon Route53, etc., have begun to increase the proportion of building with Rust, and we have also built many new services with Rust. Through a series of staffing and product enhancements, the use of Rust products in cloud services has also provided more help for enriching the community. For developers, the Amazon Cloud Technology SDK for Rust that we launched this year makes it easier for more Rust developers to use the native Rust SDK for cloud-native application-related development and practice. This is our Rust field Related series of contributions and uses.

second is the field of graph computing. Then graph computing has Apache TinkerPop in the open source field, which is an open source graph computing framework. At present, our engineers at Amazon Cloud Technology serve as the PMC chairperson in TinkerPop, which provides a lot of energy for the development of the community, and we also continue to interact with the developer community. We have engineers who have been doing various evangelistic work for the community's graph computing for several years, including Practical Gremlin online resources to help more developers use the graph computing framework Tinkerpop. At the same time, our Amazon Cloud Technology also launched a cloud fully managed service called Neptune. It is compatible with the open source Tinkerpop, and at the same time provides a very convenient ability for everyone to build graph applications in the cloud with the help of cloud services. In addition, we have also collected feedback from many developers on products and fields. We are also constantly providing feedback to our open source community through our cloud services, making bug-related fixes, and enhancing related functions to further promote the open source community through the cloud. Carry out relevant product feedback, which makes the interaction between the cloud and the open source community more and more, getting better and better.

third is the search field. Over the years, we have used a lot of open source technologies in both the Amazon e-commerce platform and the search platform. Apache Lucene is one of the most used search technologies. Now it is on Amazon’s e-commerce platform or in the voice of Alexa. In the engine, many search technologies are built using open source Lucene. Of course, in the construction of Lucene, we have actually added a lot of new features, including how to make search more effective in the complex situation facing large-scale concurrency. We also continue to update the code and change the underlying layer for Lucene at this layer, making Lucene an ideal choice for heavy work. In addition, we also provide a series of new functions to Lucene's open source platform, such as incorporating behavioral signals into search rankings, which is how to proceed after searching for a product. This has been a concern of the Lucene community for many years. Therefore, the ability that we provide to the Lucene community called the frequency of indexed custom terms has allowed us to further improve our contribution to open source and enable different developers to get better use.

fourth is the Redis field. We have contributed to Redis for many years. At present, one of the 5 Redis maintainers is working at Amazon. In Redis, we have contributed a lot of functions related to encryption. Many years ago, Redis itself did not support encryption. At that time, Amazon engineers put forward a PR, hoping to add a series of TLS to Redis, including SSL encryption. We have added these features and obtained approval, which has further improved the security of Redis. Through the combination of our open source and cloud services, more and more companies can use the further integration of open source and cloud to help them deploy Redis solutions on a large scale, such as our typical customer Airbnb, which itself has a very large Redis Cluster. However, there are actually several different processes in the process of migrating from a self-built data center to a cloud service. First, he first migrated from a local data center to the EC2 platform, hoping to use the management experience of local engineers to migrate to the cloud and use the elastic resource expansion advantages of cloud services to expand the cache architecture. But with the further expansion of the cache cluster, it found that the management and configuration operation of the cluster is a very time-consuming and labor-intensive process. Therefore, he eventually migrated all caching applications to a hosting service called ElastiCache. Through hosting services, not only can you use the advantages of many open source projects, it is fully compatible with open source, but also greatly liberates productivity. This makes the management and maintenance engineers no longer need to install, maintain, and manage the underlying architecture, which can free up energy and make the upper-level Redis application better. It is also a very important manifestation that the cloud can help customers to further improve in the open source field.

Of course, including Amazon Hudi, it is also a function of Amazon EMR in our hosting products. Our current engineers are also members of its PMC. We also make a lot of contributions in Hudi, including Hudi's guidance support, Hudi's file list and query planning, etc. Through Hudi, the ability to read S3 file lists is greatly improved.

For the robotics industry, ROS is a very important robot software. Using the development framework Amazon RoboMaker will help many developers expand ROS in the cloud, facilitating them to build better and faster robot applications. Amazon is currently a member of the ROS Technical Steering Committee and ROS 2 related working groups. And we have contributed a lot of new features, bug fixes and usability improvements in the latest version of ROS, ROS 2. At the same time, we provide more security functions, so that developers can develop robot-related products more quickly, conveniently and safely. application.

And in the Kubernetes field, we are a platinum member of CNCF and have contributed to many Kubernetes projects, such as the container network interface CNI in Kubernetes. In SIG Security, we are the second largest contributor. In addition, we have made a lot of contributions to a lot of security, observability and manageability. Kubernetes has been deployed in many production environments. However, if the deployment and construction of the Kubernetes environment are carried out in the production environment, there are actually many things to consider, not just some of the functional features and improvements that are usually more concerned, and security and management are also very important topics.

Therefore, we have greatly enhanced these security management functions in the managed service Amazon EKS, and also fed back a series of security functions to the community. This allows us to develop codes and functions that can be used by many Kubernetes customers. Get applied. This series of enhanced security and management-related updates can be used regardless of the use of Amazon Cloud Technology’s hosting services, other cloud vendors, or in the local environment. This helps customers to Better development and deployment in the production environment.

We also have a lot of contributions in other fields, including in the field of networking, automated distribution and collection, and measurement. We also have many contributions. We also have a lot of innovations in the Kubernetes field. for example,

• How to run a Kubernetes cluster on an Arm-based architecture to optimize costs
• How to deploy and call fast, machine learning models in a Kubernetes cluster
• How to perform a series of virtual machine, gpu and Kubernetes cluster compatibility plug-in interoperability

In these areas, we have also made a lot of code contributions. Through a series of continuous updates to Kubernetes, the ease of use and security in Kubernetes have been further improved, and Amazon Cloud Technology can gain a core leadership position in the Kubernetes field.

What we just talked about is the first part, how we can accelerate the use, installation, deployment and management of open source software through cloud services. Below, let's look at some open source projects initiated by Amazon.

Many of our open source projects are not conceived or invented out of thin air, they are all derived from the needs of customers. But sometimes the customer only knows the long-term goal and does not know what related open source products are needed. Therefore, Amazon will innovate on behalf of customers and launch a series of new open source projects to meet the growing demand for open source products from customers. The open source projects we initiated cover many fields, covering from databases, data analysis to machine learning, from computing to front-end development, from serverless to development tools, security, and so on. We have initiated a series of open source projects in many fields to help our developers solve more practical problems. At the same time, it also attracts developers to carry out further interaction to promote the integration of cloud computing, open source and community.

The latest project is called Babelfish for Aurora PostgreSQL. This is a new open source project released by Amazon re:Invent 2020 at the end of last year. It actually solves the problem of many customers migrating from traditional commercial databases to open source databases in the cloud. As you all know, migration is a very complicated process. In addition to migrating the underlying database, it is also best to consider that the upper-level application can be migrated at one time without changing or changing the code. Babelfish is such an open source product. Babelfish provides a transparent conversion layer to help applications migrate to PostgreSQL easily, helping customers use open source databases better and faster in the cloud or even locally to reduce costs, improve ease of use, and contribute to the open source community. Make a brand new contribution. This open source project will be officially launched on Github in the second half of this year, in addition to some other projects.

first JDK Amazon Corretto launched by Amazon. well known that Java currently has a series of problems in the licensing area. So we launched the Amazon Corretto product to participate in the OpenJDK community. We are currently also a leader and partner in the OpenJDK community. On top of Amazon Corretto, we also provide a lot of new capabilities, including increasing the performance of encrypted things on Java, optimizing the Arm architecture to help developers use open source JDK to build better commercial applications, or open source applications, and so on. At the same time, we are constantly providing unlimited free security patches for Amazon Corretto. We also have a lot of investment in the Java field. James Gosling, the father of Java, is currently an outstanding engineer at Amazon Cloud Technology, responsible for many product-related developments. Including Corretto, and even including a series of open source software such as Greengrass 2.0, which was also released at the end of last year, provided us with invaluable guidance for a series of open source development.

In fact, for open source projects, personnel are always a very important factor. We at Amazon Cloud Technology also attach great importance to recruiting talents in the open source field. We have many engineers and partners who are very, very good open source experts and talents.

Another open source project is Firecracker. It is an open source virtualization technology based on the Apache2.0 license, compatible with multiple capabilities. One is the containerized security management capability, and the other is the speed and performance of the container. Before Firecracker, in the container field, it was actually very difficult to be compatible with both capabilities. With Firecracker, you can take into account both aspects, providing very good help for container virtualization, including serverless applications. Just talked about our current cloud services, and we also use many Firecracker-related technologies, so that everyone can directly use our open source products to build applications.

For Firecracker, we don't just treat it as an open source project, but hope to create a complete ecosystem including developers and partners to facilitate the community and developers to use it quickly and conveniently. For example, in North America we have a partner called Weaveworks. It is also a well-known startup company in the container and serverless fields. They have done a lot of development based on Amazon's open source products. For example, he used the developed eksctl tool, which is currently the official interface of our Amazon EKS; built an Ignite product based on Firecracker to further develop and develop serverless and containerization. management. At the same time, through ecological cooperation, the scope of use of developers has been expanded, so that our open source products have better iterations, and at the same time continue to expand the circle of friends.

There is also our Bottlerocket in the field of Linux operating system. uses Bottlerocket, a specially constructed Linux open source operating system, to further enhance the security and flexibility of the system, and to help developers deal with different tasks. In the Linux field, we have contributed many open source products. In addition to Bottlerocket, there are also Nitro Enclaves, which are used for more secure virtualized isolation, and are also provided for various industries, such as the financial industry and other typical data-sensitive fields. A better security experience. It can perform hypervisor management of virtual machines across multiple operating systems and processor architectures. At the same time, it establishes open standards and interfaces, and can build standard interfaces in a series of environments to rebuild and deploy the environment.

We also hope to provide developers with more tools. Based on the previous launch of CloudFormation, a brand-new tool called CDK, Cloud Development Kit was launched. It is 100% open source and can help developers use infrastructure as code for dynamic deployment and reusable resources. Through a series of tools, developers can freely view and distribute related code. At the same time, this project is actually contributed by many engineers, especially engineers from the community. 48% of the project’s contribution is community-driven, making the product recognized by many community partners from the beginning of its design. It also makes the product very well recognized by many developers.

The other is OpenSearch. This open source product was actually launched just last month and is an open source search and analysis suite based on Apache2.0. Because everyone knows that Elasticsearch changed its license at the beginning of this year, from Apache2.0 to SSPL. This pair of developers use Elasticsearch in the cloud. In fact, it provides a very unfriendly protocol experience. So we decided to re-use Apache2.0 to build a set of open source software. Therefore, we Fork Elasticsearch and continue to use Apache2.0 for related open source distribution and deployment. At the same time, we also added a lot of new functions and core capabilities. At present, OpenSearch has been officially open sourced, and we hope that our community and developers can participate in providing a better growth atmosphere and environment for a more open search platform.

For a series of observability, we also released our own release, Open Distro for OpenTelemtry. To further increase the observability of the upstream, a series of related functions are added, and a series of API and SDK specifications are provided. At the same time, it provides numerous contributions to upstream.

Amazon Cloud Technology is the main partner of CNCF, cooperating on many levels, including Kubernetes, Prometheus, Envoy, and so on. At the same time, we also hope to bring more CNCF open source projects to the cloud environment to provide developers with more possibilities and convenience. At the same time, we have also made a lot of contributions to CNCF projects, including etcd nanny, cdk8s and many more. At the same time, we are also continuing to contribute to some CNCF upstream projects. I just talked about it, including many contributions to Kubernetes.

Amazon Cloud Technology has made a lot of contributions in the open source field, and our driving force actually comes from the end customer. We uphold the concept of customer first. Customers want to use open source software quickly and conveniently in the cloud environment, and customers want product innovation in the open source field. Therefore, we innovate on behalf of our customers to initiate more open source projects and more open source cloud services. Customer-oriented needs allow us to get closer to the market, listen to the voice of developers, and develop better cloud products and open source projects to give back to everyone.

We have a lot of open source contributors internally. Here I will give a few examples as a representative. You can see that many of our open source contributors are actually female developers. Through the efforts of a series of developers, Amazon's open source contributions have become more and more large, and it has also attracted more developers to participate and contribute to the community.

In addition, we are constantly helping the open source community to grow. Drupal is a very popular community. It currently supports more than 550,000 websites, has more than 1.3 million communities and many developers. It is currently built on the platform of Amazon Cloud Technology, and we have also contributed a lot to the thriving development of the Drupal community. Recalling more than 20 years ago, when I started my business, it was built using open source MySQL, but as the number of users continued to increase, he found that it was very difficult to expand MySQL. There are many options that need to be adjusted, the communication between many nodes, and many expansion details need to be considered. Therefore, in the process of continuous development and practice, they began to use the services of Amazon Cloud Technology. Through the fast and elastic characteristics of cloud services, they can greatly save costs, and at the same time, they can increase elastic capacity, making cloud services and open source software strong and powerful. It is an ecological platform for huge developers with millions of users, communities and tens of thousands of user participants.

We are also constantly helping open source companies to grow further. Everyone knows that Amazon Cloud Technology has a third-party market called Marketplace. Many third-party open source companies can put their open source products online on our Marketplace. We can work closely with open source companies to help them further invest in the market. Because whether it is a commercial software company or an open source software company, especially the operating company behind the open source software, there must be a means of profit. Therefore, we hope to make the open source market bigger and stronger, and share the results in this area with many partners. Good cooperation can be carried out through Marketplace, making the powerful combination of cloud and open source more possible.

Of course, we also help many customers to build a series of good practices for open source projects. For example, Lyft, as we all know, he has also done a lot of work related to autonomous driving. Behind it is the envoy platform built on the platform of Amazon Cloud Technology. Amazon engineers have done a lot of cooperation with them to help them how to quickly expand the envoy platform on EC2, including the Kubernetes platform. We have made a lot of contributions with Lyft engineers, including NETFLIX, which will have open source Spinnaker products. In the cooperation with NETFLIX, our engineers have also constantly discovered errors in Spinnaker, so our engineers have also participated in this open source project and made a lot of code contributions and performance improvements.

We also have a lot of financial support to help various foundations support open source and community growth. In addition, we also hope to have more products to help customers use open source. Migrate from traditional commercial-licensed software to software that uses open source licenses. For example, the Database Migration Service service helps to migrate to open source databases and so on. Regardless of whether you use open source cloud services or build open source on a local infrastructure, we have provided relevant tools to help you move quickly towards open source.

We also provide a series of stages for developers and the community. Amazon Cloud Technology also has developer expert projects, with two levels, first is Hero , currently we have relatively few Heroes in China. We also hope to cooperate with many developers to quickly build an ecosystem in China. If you are good at technology, in a certain field, and hope to give public lectures on more occasions on how to use cloud service products, to build open source or use open source to implement solutions, you can sign up for our project. Continuously increasing influence in the event. If you think Hero still has some difficulties, then we have second project, called the Builder Community Community . Builders conduct extensive exchanges and interactions with everyone in the community, share knowledge for everyone, provide guidance on relevant technical resources, and drive the vigorous development of the community.

Regarding cloud services and open source, in fact, everyone can see the current trend of integration and development. Cloud computing has further promoted the development of open source, and open source has also enabled more and more developers to use cloud computing to build applications. Cloud and open source are actually a non-zero-sum game. Our common hope is to make the market bigger and stronger, so that more open source projects, open source vendors, and more cloud service providers can be in the larger and larger market. Can find a suitable position in both, which can provide developers with more choices. Therefore, we also hope to work with more Chinese developers to make open source better and stronger.

Finally, I want to say that open source is not only Chinese, it is also global. I hope that developers can stand in a global perspective and look at open source and contribute to open source. We hope that through our technological influence, developers all over the world can benefit from the contributions of Chinese developers. We also hope that more and more Chinese developers can expand their influence globally. .

thank you all.