Integrated cloud (Unified IaaS), future-oriented IT infrastructure architecture choice

Author introduction: Qiu Jian, ), Tsinghua Pengshuo, former Meituan Cloud architect and technical core, led the team to complete Meituan Cloud’s earliest technical selection, architecture design and code delivery.

With the advent of the digital age, IT systems have become an indispensable part of the normal operation of human society. In the not-too-distant future, technologies such as intelligent manufacturing, 5G and artificial intelligence will become important engines for the development of productivity, and human society will face an unprecedented wave of comprehensive and thorough digitalization. As the platform and carrier of IT system operation, IT infrastructure is the cornerstone of digitalization. In this wave of digitalization, companies must actively embrace cloud computing technology and adopt a future-oriented IT infrastructure that is in line with technological development trends in order to win opportunities in the future competition.

1. Cloud computing development trend judgment after more than ten years

Cloud computing technology started from the launch of AWS's first public cloud service S3 in 2006. Today, in 2019, some patterns and trends have gradually become clear:

First of all, private cloud is still the IT infrastructure of choice for large and medium-sized enterprises and some sub-sectors, such as government affairs, finance, healthcare, education, energy, and manufacturing. As major public cloud vendors have successively launched their public cloud expansion plans for the privatization of government and enterprise customers, such as AWS Outposts, Azure Stack, Google Anthos, and domestic Alibaba Cloud, Tencent Cloud and other private cloud/proprietary cloud deployment solutions , "Whether private clouds will gradually die out with the development of public clouds" has been rejected by public cloud vendors themselves. Facts have proved that private clouds will continue to exist for a long time, will coexist with public clouds, and become an important part of enterprise IT infrastructure.

Second, the public cloud continues to develop rapidly and gradually becomes the main provider of enterprise IT infrastructure. In Q3 2018, cloud hardware expenditures accounted for 50.9% of total IT revenue. In 2018, China's private cloud infrastructure spending was US$3.8 billion, and public cloud infrastructure spending reached US$8.2 billion (source: IDC). Therefore, the public cloud has become the most important provider of IT infrastructure. Especially for small and medium-sized enterprises, their IT infrastructure may be completely built on top of the public cloud. At the same time, some large-scale enterprises in technologically leading industries, such as the Internet, finance, and manufacturing, have also begun to use public clouds and explore hybrid cloud architectures that combine the advantages of public and private clouds.

At the same time, competition in the public cloud market is extremely fierce. In the end, only a few players with strong technology and capital will enter the stage of oligopoly market. On the one hand, the products and services provided by public cloud vendors are essentially the same. They are all IT infrastructure resources and software services on them. On the other hand, each vendor strives to improve its own products, enrich product lines, and make special features to obtain Competitive advantage, attract incremental users, and avoid the loss of existing users. Therefore, in the end, the product function matrix provided by the public cloud is basically the same, but each has its own merits and differences in characteristic functions, regional coverage, and user experience. As users become more dependent on public cloud products and services, service switching and migration between public clouds will become more and more difficult, and there is an invisible gap between cloud and cloud. Of course, the public cloud entering the oligopoly stage also means that the list of public cloud providers will remain relatively stable for a long time, which means that the adaptation costs for all public cloud APIs will become controllable and feasible.

Another trend that cannot be ignored is that Kubernetes has become the de facto standard for container orchestration and has gradually become the standard environment for application deployment and operation in the cloud-native era. With the gradual improvement of Kubernetes' storage and network support, not only stateless services can be deployed and run on Kubernetes, but stateful data storage services can also be run on Kubernetes. At the same time, based on Kubernetes, a prosperous and powerful open source software ecosystem and a complete tool chain have been developed. For example, Helm realizes the automatic deployment of software suites, Operator realizes the automatic operation and maintenance of software, and lstio provides microservices RPC communication management architecture. Knative provides Serverless running framework and so on. It is foreseeable that Kubernetes will become the standard runtime environment for distributed applications in the future and become the "Linux" in the era of distributed applications. A software ecosystem dominated by open source software will be built on top of Kubernetes, which includes not only application software, but also various PaaS middleware, such as message middleware, various open source databases, development frameworks, AI training frameworks, etc., which are truly realized The vision of "Open Source Rule the World". Based on this trend judgment, major public cloud vendors have successively launched their own Kubernetes solutions, allowing native Kubernetes to run more efficiently on their own cloud platforms.

2. The determination and uncertainty of the enterprise's future IT infrastructure

Based on these facts and trends, we can imagine the future of enterprise IT infrastructure will be like this:

First of all, hybrid cloud architecture is the best choice for enterprises.

In the future IT infrastructure solutions of enterprises, public cloud and private cloud will no longer be an alternative, but two inevitable components of a complete IT infrastructure. On the one hand, companies may have their own private clouds, but there are also companies that run entirely on public clouds. On the other hand, enterprises will inevitably use public clouds, and the public cloud resources they purchase will become part of their private IT infrastructure.

Secondly, Kubernetes will become the standard operating environment for enterprise cloud native applications.

Just like today's enterprise applications are running in Linux, future enterprise services will be cloud-native, distributed, and run in Kubernetes. Enterprises will have several Kubernetes clusters, running different applications, distributed on different infrastructures, some running on the local IDC, some running on the private cloud, and some running on the public cloud.

The above two points are generally recognized as relatively definite conclusions, but there are many other issues that currently have no definite conclusions, such as:

1. Although the use of public cloud is an inevitable choice for enterprises, enterprises will decide whether to use multiple public clouds or a single public cloud. There are many reasons for adopting the multi-public cloud solution, and the benefits are obvious, such as avoiding supplier lock-in, improving bargaining power, and obtaining richer features and geographic options. But at the same time, it is difficult to use multiple public cloud resources for unified management, and the high cost of service switching and migration between clouds prevents users from choosing multiple public clouds.

2. Although cloud computing technology has been developed for more than ten years, a large proportion of the local IT infrastructure of enterprises is not cloudified, neither managed through private cloud nor even virtualized. Although the future cloud-native applications will run in the Kubernetes container environment, there are still many traditional applications that are not containerized. Moreover, to be realistic, for most companies, perhaps for a long time in the future, they will still be dominated by non-cloud native traditional applications. Therefore, the future IT infrastructure of the enterprise cannot simply be assumed to run Kubernetes in a normalized manner, but should provide the virtual machine or bare metal environment required for the operation of these traditional applications. Does this kind of enterprise cloud transformation process still need to go through the long path of private cloud-hybrid cloud-multi-cloud, and then deploy a set of private clouds to realize the cloudification of local IT infrastructure?

3. On the one hand, with the development of business and industry drivers, enterprises' requirements for IT infrastructure, whether in terms of scale, efficiency or stability, will become more and more demanding. Agile development and DevOps will become standard equipment for enterprises. On the other hand, with the development of technology, enterprise IT infrastructure will become more complex and difficult to control. Enterprise IT resources will not only be physical servers, but also virtual machines and containers. In addition to x86, there will also be minicomputers, ARM, and even heterogeneous computing resources such as GPU, FPGA, and TPU. There are also multiple technology options for networking and storage. At the same time, as of today, only mainstream public cloud providers provide thousands of cloud products and services in more than 500 availability zones in more than 200 regions around the world. As long as the enterprise is willing, a global-scale IT infrastructure is readily available. How do enterprise IT personnel deal with the challenges of IT infrastructure in terms of scale, efficiency and complexity?

4. Even if the future enterprise IT infrastructure will converge to fully run on Kubernetes, a single Kubernetes cluster is often only used for a single specific purpose, such as testing or production clusters in a specific department. It is normal for an enterprise to have multiple Kubernetes clusters. Managing multiple Kubernetes clusters, especially multiple Kubernetes clusters deployed in a multi-cloud environment, is still a problem. Although Kubernetes shields the differences in the underlying infrastructure and provides a consistent interface and operating environment upwards, the management interfaces and network storage solutions of Kubernetes in various public clouds and local IDCs are not unified. When creating, expanding, and adjusting the configuration of a Kubernetes cluster , Still facing the problem of docking multiple supplier interfaces. At the same time, the Kubernetes clusters distributed on multiple public clouds are not connected, not only the control information cannot be synchronized, but the data link level is isolated from each other and is an island of each other. Therefore, the linkage of multiple clusters cannot be realized, and the switching and coordination between the clusters cannot be realized. The Kubernetes cluster solution in a multi-cloud environment is still to be explored.

5. With the improvement of the Kubernetes ecosystem, users will have two choices when using PaaS services on public clouds: use PaaS services provided by public clouds or cloud-native open source PaaS services based on Kubernetes. The former has a high degree of productization, is easier to use, and can get commercial support. But there are also problems of being locked in by commercial products, difficult to switch, and expensive to use. Using the latter requires a certain degree of control over open source software, but the price is cheap (the fee for using the cloud host), based on open source technology, has strong community support, and the architecture is open, flexible and easy to expand.

Third, the integrated cloud (Unified IaaS), future-oriented IT infrastructure architecture choice

In response to the above deterministic conclusions and uncertain questions, our answer is that the best choice for future-oriented IT infrastructure management is the unified cloud (Unified IaaS). As the name implies, the so-called converged cloud is to converge and manage all IT infrastructures distributed in a multi-cloud environment (local IDC, private cloud and public cloud) to build a "cloud on cloud" converged IaaS platform. Converged cloud is essentially a private cloud, but the scope of IT resources managed is no longer limited to local IDC, but also includes IaaS resources purchased by enterprises in public clouds. For enterprises with a pure public cloud architecture, the integrated cloud management is entirely the public cloud resources purchased by the enterprise. The difference between converged cloud and traditional cloud platform is not in the scope of resources to be managed, but in response to the above-mentioned development trends and problems of enterprise IT. In terms of design concepts, converged cloud and traditional cloud platforms have the following differences:

First of all, Converged Cloud is geared towards a multi-cloud environment.

In the deployment scenario of the converged cloud, the IT infrastructure of enterprise users not only includes the part deployed in the local IDC, but also the part purchased by the user in the public cloud. Converged Cloud manages all the IT infrastructure of the enterprise through one platform. The first is the unification and integration of the management plane to achieve unified API access for private and public cloud resources, not only to achieve resource management, but also to unify billing and resource management. Allowing users to call across clouds is as convenient as using a cloud platform. The second is the connection of the data plane. Through the integration with the cross-cloud network solution, the coordination of the control plane and the data platform is realized, and the inter-cloud intranet intercommunication of the entire platform is achieved. In addition, Converged Cloud will also provide tools for cross-cloud data migration to facilitate users to implement cross-cloud application migration. In short, the goal of converged cloud is to fill the gap between cloud and cloud, reduce the cost of cross-cloud switching and migration, and make multi-cloud deployment easier.

160d724dbd91e7 Secondly, convergent cloud realizes the comprehensive

Converged cloud can not only manage the private cloud and public cloud resources that have been clouded, but also needs to have built-in private cloud technology to manage bare metal cloud, virtualization technologies such as KVM and VMware ESXi, and computing resources such as ARM and GPU. For enterprises that have not yet deployed a private cloud, deploy converged cloud to realize the private cloud of enterprise private IT infrastructure in one step, and realize cloud management of bare metal, KVM, VMware ESXi, GPU, etc., without the need to introduce additional private The cloud solution reduces the implementation cost and management complexity of enterprises going to the cloud.

Third, intelligence will be the core feature of Converged Cloud.

On the one hand, Converged Cloud optimizes the scheduling strategy of IT resource allocation, finds out idle and wasted IT resources, and improves the utilization of IT resources. On the other hand, predict resource requirements in advance and discover hidden dangers of system failures to ensure the smooth operation and expansion of the system. Through data and algorithms, IT infrastructure is made more intelligent, helping enterprise IT personnel to manage the challenges of future IT infrastructure in terms of scale, efficiency and complexity.

Fourth, the converged cloud is geared towards Kubernetes.

Converged cloud, on the one hand, realizes the unification and integration of Kubernetes underlying infrastructure in a multi-cloud environment. One is to provide Kubernetes with a unified IaaS interface in a multi-cloud environment through a unified API, which facilitates the deployment of a Kubernetes environment across clouds. The second is to open up the intranet of cross-cloud Kubernetes in the data plane to realize cross-cloud communication. On the other hand, it directly provides a unified Kubernetes cluster management control API and a synchronization mechanism for cluster information to achieve unified management and control of cross-cluster Kubernetes, and realize the synchronization and unification of accounts, permissions, and configurations across Kubernetes clusters.

Finally, Fusion Cloud fully embraces open source technology.

The course of software development shows that the future of PaaS is open source. Suppliers cannot meet all the PaaS needs of enterprise customers on their own. Therefore, Converged Cloud focuses on the unified management of computing, network and storage IaaS resources distributed by enterprises in local IDC and public clouds, providing reliable underlying infrastructure for multi-cloud Kubernetes, and software and application requirements on Kubernetes rely on open source ecosystems to provide solution. Convergence cloud users' needs for PaaS provide services to users through the Kubernetes application market and integrate open source PaaS applications. On the one hand, it lowers the technical threshold for users to use open source PaaS. On the other hand, it relies on a strong open source community to provide users with open, flexible and rich software products, avoiding private PaaS software from locking users.

Based on the above assumptions, the architecture of the converged cloud is as follows.

Downward: Convergent cloud unified management of multi-cloud infrastructure, mainly to achieve unified management of IaaS resources such as computing, network, and storage in a multi-cloud environment. For local IDC's unclouded resources, mainly bare metal, KVM virtual machine (Libvirt), VMware ESXi virtual machine (vSphere), cloud management is realized through the built-in private cloud solution. For private cloud and public cloud resources, unified management is achieved through APIs.

Upward: On the one hand, the converged cloud provides a complete operating system runtime environment for traditional applications through virtual machines and bare metal; on the other hand, it provides a multi-cloud operating environment for Kubernetes, and unified management of multi-cloud Kubernetes. On top of Kubernetes, it provides a container runtime environment for cloud-native applications. At the same time, PaaS middleware services are provided based on Kubernetes and open source components.

In short, convergent cloud manages multi-cloud IaaS resources uniformly downwards; provides multi-cloud support for Kubernetes upwards, and meets enterprise PaaS needs through open source ecology; other needs of users can be obtained by accessing native services of public cloud, thus fully satisfying the future enterprise IT foundation Multi-level requirements for facilities.

With the popularization of big data and artificial intelligence technologies and the advent of the 5G era, IT infrastructure has become more and more important, becoming the cornerstone of enterprises' digital transformation and fully embracing the digital age. Based on the multi-cloud trend of enterprise IT architecture, convergent cloud emerged. Converged cloud is a cloud platform for future-oriented enterprise IT infrastructure management. It is designed for the problems of enterprise IT infrastructure in the future and will help enterprises meet the challenges of upcoming digital transformation.