[CDS Technology Demystification Series] Alibaba Cloud's cloud-defined storage is here

Introduction to newly released cloud-defined storage CDS of 1615ad7b0ebb98 and traditional storage arrays, distributed storage, and software-defined storage? How does the Alibaba Cloud storage team view the development trend of storage in the future? This article invites the core technical leader of the CDS R&D team to unravel the mysteries surrounding Alibaba Cloud CDS.

Cloud Defined Storage (CDS: Cloud Defined Storage) is officially released

On the occasion of the Mid-Autumn Festival and National Day of 2021, the Alibaba Cloud storage team released a new cloud-defined storage CDS.

The CDS 1.0 version released this time, based on the Alibaba Cloud Feitian operating system and the distributed storage base Pangu, is the first in the industry to simultaneously support the object storage service OSS, block storage service EBS, log service SLS and disaster recovery service HBR, especially through Pangu's new The QoS framework supports the integrated deployment of OSS object storage services and EBS services of block storage services on the same set of physical servers. The storage service released this time has basically the same code version as the large-scale storage service on the public cloud, bridging the problem of the large gap between the previous hybrid cloud version and the public cloud version, allowing customers to enjoy basically the same functions as the public cloud .

Why does the Alibaba Cloud storage team release CDS? What is the difference between CDS and current traditional storage arrays, distributed storage, and software-defined storage? How does the Alibaba Cloud storage team see the development trend of storage in the future? Here, the core technical leaders of our CDS R&D team will reveal the mysteries surrounding Alibaba Cloud CDS through a series of technical articles.

Why develop cloud-defined storage?

Business needs

Since Alibaba Cloud Storage established the HCS (Hybrid Cloud Storage) department at the end of 2016, internal and external businesses have always had a lot of demand for distributed storage products with flexible scale configuration and competitive functions. At that time, comprehensively considering the investment in product research and development, Alibaba Cloud still lacked experience in the hybrid cloud market, as well as many factors such as the number of participating vendors in the domestic market and fierce competition. The disaster recovery all-in-one machine and other integration start, first enter the different segments of the traditional storage market to test the water, to understand and explore this market for Alibaba Cloud, and reserve experience and ammunition.

Since 2018, with the accelerating pace of IT infrastructure cloudification, cloud-based infrastructure or data centers have become the first choice for government and enterprise customers, bringing a huge market to new storage vendors, especially software-defined storage vendors Opportunities. At the same time, with the development of Alibaba Cloud in the hybrid cloud market, the requirements for the output of Alibaba Cloud storage in the hybrid cloud scene are getting higher and higher, so the storage team officially started the planning, design and development of related products in 2019.

Difficult decision

After focusing on the direction of distributed storage and SDS, through analysis and research, we found that the homogeneity of domestic SDS products is very serious. If we break away from the public cloud technology stack and start a similar product, it will be difficult to build a durable core. Competitiveness. From the perspective of technology and engineering, the parallel development of public cloud and proprietary cloud product architectures will cause a huge waste of resources. Many similar features need to be reworked on different products, and they cannot be guaranteed due to architectural constraints. Can support it. Finally, we decided to start the self-research of hybrid cloud distributed storage SDS. Technically, we insist that public cloud and hybrid cloud CDS share the same set of technologies and codes to maintain the flexibility, scalability, and differentiated competitiveness of service hosting of public cloud storage services. Actively complement the advanced features of enterprise storage. Therefore, the architecture, functions, and user experience of the hybrid cloud CDS are consistent with the public cloud, which is the origin of the "cloud definition" of cloud-defined storage.

Next, we will introduce the characteristics of software-defined storage and the difference between cloud-defined storage and software-defined storage in turn.

Software-defined storage

The origin of software-defined storage

Software Defined Storage (SDS: Software Defined Storage) was first seen in 2012 when VMware and EMC promoted a series of market battles in the Software Defined Data Center (SDDC: Software Defined Data Center), and subsequent data center/server/storage vendors followed suit. After entering this definition, everyone's strategy is to respond to the increasingly obvious demands of enterprises to go to the cloud, as well as the pressure and competition brought by AWS/Azure cloud computing vendors, so they redefine the path of enterprise data centers and storage for cloud computing. And direction. The hottest part of the history of SDS that year is shown in the figure below:

Key features of software-defined storage

Software-defined storage has many characteristics, and the industry has three generally recognized characteristics: abstract, pool, and automate. The driving force behind these characteristics is enterprise cloudification and enterprise ICT infrastructure cloudification.

• Abstraction, decoupling of software and hardware, decoupling of management and control: cloud computing vendors and cloudized data centers use a large number of standardized/commercialized servers to replace proprietary hardware storage devices such as EMC/NetApp/IBM, and large-scale procurement and storage/ Separate bidding for servers promotes the decoupling of storage software and hardware. The decoupling of data plane and control plane also helps customers manage different hardware (data plane) of different vendors through a unified control plane.
• Pooling and horizontal expansion: In order to improve resource utilization efficiency, storage virtualization technology is used to pool various storage and data chimneys.
• Automation: Through the development of API, the software can control the addition, deletion, modification, and checking of storage resources and services, which is more efficient than the human-machine interface and has a stronger business elasticity.

The difference between software-defined storage and traditional storage arrays/software

SDS and traditional storage arrays are different in terms of hardware coupling, node/controller expansion method and scale, interaction and management interface:

The difference worth mentioning is that the "software definition" of SDS abroad emphasizes that SDS is rich in API/SDK and can be driven by software programming, while traditional storage has more human-computer interaction interfaces and is mainly driven by administrators.

What is cloud-defined storage

Cloud-defined storage CDS is a brand new software-defined storage, which can optimize the output of software and hardware on a customized CDS storage server, and also has a software output mode that supports X86 to various Xinchuang platforms. The core key word among them is "cloud definition". In summary, there are six major capabilities:

• Cloud Native

CDS is not a re-developed product. It is the introduction and release of various storage services of Alibaba Cloud Public Cloud to a unified CDS storage platform in accordance with product planning and market demand. The storage services in the internal CDS, such as OSS codes and features, The public cloud is completely consistent; to the outside, the various cloud storage services integrated by CDS from the console (GUI), command line (CLI) to Open API/SDK are consistent with the public cloud, reducing the use of original Alibaba Cloud customers and partners And the difficulty of integration.

At the same time, CDS also naturally supports the hierarchical architecture of public clouds: cloud-region (Region)-availability zone (AZ)-cluster (Cluster).

• Cloud Scale

Compared with most distributed storage or SDS vendors, the biggest advantage of Alibaba Cloud CDS is the CDS. These codes have been polished and verified by Alibaba Cloud public cloud millions of customers and various businesses of Alibaba Group (such as Double Eleven) , Supporting the pressure of ultra-large-scale and complex businesses. Taking CDS's OSS storage service as an example, a single bucket can support trillions of objects in the production environment, reaching EB-level capacity.

• Cloud Service

CDS provides multiple cloud storage services (generally the most common block storage and object storage services) on the same storage platform, and supports multi-tenancy. The front desk can be self-service (Self Service) like public cloud, the back-end can be operated and maintained by customers independently or managed (locally or remotely), and various basic storage services are managed and controlled through the CDS cloud management suite. Pangu unified data base, at the same time, you can choose to open advanced storage services such as log service, disaster recovery service, data management and analysis service, etc.

At the same time, in order to accelerate the cloud migration of enterprises, Alibaba Cloud Storage simultaneously released enterprise storage functions on public clouds and CDS, such as EBS-based enterprise block storage ESSD, and HBR-based disaster recovery and backup functions.

• To Cloud

The stored data flows between the CDS and the cloud outside the CDS in a controlled and secure flow through the network.

• Cloud deployment (on Cloud)

CDS is directly deployed in the cloud environment as a cloud storage service to provide external services. The cloud environment here can be various cloud architectures such as public cloud, local cloud, edge cloud, and proprietary cloud.

• Hybrid Multi Cloud

In addition to the needs of external customers, there are also many business units within Alibaba Group that have storage requirements when exporting products or services. Combining these requirements, they or their customers have a desire to avoid being locked in by a certain supplier. Hybrid multi-cloud has become standard. Here, there are two different interpretations of hybrid multi-cloud: one is Alibaba Cloud's hybrid multi-cloud, including Alibaba Cloud public cloud, hybrid cloud, local cloud, edge cloud computing, etc.; the other is Alibaba Cloud and other cloud computing services business.

Based on the unified Pangu 2.0 distributed storage hybrid multi-cloud, CDS realizes multi-state cloud deployment such as hybrid multi-cloud and cloud connectivity:

Redefined feature release model

Alibaba Cloud CDS not only redefines the new baseline of hybrid cloud SDS, but also redefines the engineering model of Alibaba Cloud storage product and feature research and development. Starting from CDS 1.0, cloud storage service items that have been released on the CDS platform will be required Public cloud and hybrid cloud scenarios will be released quasi-simultaneously. At the same time, hybrid cloud output, especially in scenarios where there is no public network connection, depends on the technical capabilities of customers or field service personnel; Alibaba Cloud CDS products focus on polishing intelligent operation and maintenance, RAS, and automatic diagnosis capabilities, and these capabilities also feed back In public cloud storage services, improve system operation and maintenance efficiency and enhance system stability.

CDS's redefined engineering model has greatly accelerated the energy efficiency of the hybrid cloud storage version. In the next version of CDS this fiscal year, important products and features such as cloud storage gateways, new enterprise-level fast storage ESSDs, and file storage NAS will be quickly released. roll out.

Outlook

Alibaba Cloud CDS is a cloud-defined storage platform. The current CDS 1.0 version supports a variety of Alibaba Cloud storage products and data services (including block storage EBS, object storage OSS, log service SLS, and disaster recovery service HBR). In the next technical series, we will gradually introduce the features of various products and key modules such as platform and control services, Pangu, EBS, OSS, SLS, etc. in CDS.

The release of Alibaba Cloud CDS1.0 represents that Alibaba Cloud Storage has begun to enter the hybrid cloud storage market in the form of an authentic cloud storage to meet the growing demand for cloud computing. As a domestic public cloud leader and a challenger in the hybrid cloud market, we have firm confidence and determination to promote the development of China's storage technology together with experts from universities and friends and business peers.

Original work: Alibaba Cloud Storage Cihu

Series articles pass the door:

1. Cloud’s cloud-defined storage is here 1615ad7b0ec330 https://developer.aliyun.com/article/792044?spm=a2c6h.13148508.0.0.3eef4f0ecyZOjQ
2. Cloud CDS-OSS Disaster Recovery Big Secret 1615ad7b0ec3a8 https://developer.aliyun.com/article/792000?spm=a2c6h.13148508.0.0.3eef4f0ecyZOjQ

Copyright Notice: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.