Introduction to Fluid is an efficient support platform for data-intensive applications in the cloud native environment under the CNCF Foundation. It was jointly initiated by Nanjing University, the Alibaba Cloud cloud native team, and the Alluxio open source community. Since its open source release, the project has attracted the attention of many experts and engineers in related fields, and the community has continued to evolve under the positive feedback from everyone. Recently, Fluid 0.6 version was officially released. In this version, Fluid mainly added and improved three aspects.
Author | Gu Rong Fluid Open Source Community Chairperson Nanjing University PASALab
Fluid is an efficient support platform for data-intensive applications in the cloud native environment under the CNCF Foundation. It was jointly initiated by Nanjing University, the Alibaba Cloud cloud native team, and the Alluxio open source community. Since its open source release, the project has attracted the attention of many experts and engineers in related fields, and the community has continued to evolve under the positive feedback from everyone. Recently, Fluid 0.6 was officially released. In this version, Fluid mainly added and improved the following three aspects:
- Data-aware Pod scheduling supports coordinated scheduling of data and applications, strengthens ecological integration, and adapts to more open source schedulers.
- Rich data set operation functions, support automatic elastic expansion and contraction of data sets, and dynamic update of mount points.
- New and enhanced cache engines, support for high availability of cache engines, and new public cloud cache engines.
Fluid open source project address: https://github.com/fluid-cloudnative/fluid
The new official website of the Fluid project: https://fluid-cloudnative.github.io/
The development requirements for these three main functions come from the actual production feedback of many community users. In addition, Fluid v0.6 has also carried out some bug fixes and document updates. Welcome to experience Fluid v0.6!
download link: 16140417229488 https://github.com/fluid-cloudnative/fluid/releases
The following is a further introduction to the functions of this new version release.
Data perceives Pod scheduling optimization
Fluid has been committed to using cloud-native resource scheduling capabilities to meet the needs of efficient operation of data-intensive applications on the cloud. In the last version of Fluid v0.5, we have made a series of optimizations in data set scheduling for the lack of architecture awareness in data set scheduling. In Fluid v0.6, we further aimed at the problem of lack of data awareness in container orchestration, and designed and realized the optimization of Pod scheduling.
The optimization scenarios that we are currently considering include:
- When using non-K8s native scheduler, it is not compatible with Fluid well
- When using the Pod scheduling of the data set, the consideration of cache distribution is not considered
- Pods that do not use data sets may be scheduled to nodes with cache, which will affect other Pods to use cache
We formulate the Pod scheduling strategy based on the data set cache location information, and inject the generated scheduling information into the Pod through the webhook mechanism, and finally achieve the following functions:
- In the default Fuse mode (Fuse and cache engine are deployed on the same node), it supports K8s native scheduler, and third-party schedulers such as Volcano and Yunikorn to achieve Pod data affinity scheduling
- In the global Fuse mode, the Pod is first scheduled to nodes with data caching capabilities
- When the Pod does not use the data set, try to avoid scheduling the Pod to nodes with cache
For how to enable and configure the Pod scheduling optimization function, please refer to the sample document on Github.
Rich data set operation function
1. Data set online elastic cache expansion and contraction
Fluid v0.5 opened the way for online elastic expansion and contraction. At that time, it provided the ability to manually expand and contract online. However, in a real production environment, manual scaling has greater complexity and delay. Automatic elastic scaling is one of the core capabilities of Kubernetes, which has been around this stateless application load. Now, stateful data-intensive applications can also use the elastic scalability of the distributed cache provided by Fluid to achieve flexible expansion and contraction of the data cache. Based on Runtime, Fluid provides performance indicators such as cache space and existing cache ratio, combined with its own expansion and contraction capabilities for Runtime resources, so as to achieve the on-demand scalability of data caches.
Furthermore, we found that triggering automatic data caching capacity elastic expansion and contraction based on the proportion of data caching volume has many advantages, but there is also a shortcoming that needs to be adjusted according to the resource pressure after calculating a reasonable value, which has a certain degree of lag. sex. Therefore, Fluid v0.6 combines CronHPA to provide the ability to expand and shrink regularly, so as to realize the on-time expansion and shrinkage of the data cache according to the time characteristics of the application's own use of data, and make full use of the cluster computing and storage resources to accelerate the data access performance of the application. . At present, the use of automatic expansion + timing reduction can maximize the Fluid platform into a controllable elastic cache resource in the K8s cluster. Currently, Fluid's Alluxio Runtime provides complete support in this regard.
2. Data set mount point dynamic update function
In each data set of Fluid, a number of mount points (mountPoint) are declared. For example, in the following data set named test, two mount points, hbase and spark, are declared:
apiVersion: data.fluid.io/v1alpha1kind: Datasetmetadata:name: testspec:mounts: - mountPoint: https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/stable/ name: hbase- mountPoint: https://mirrors.tuna.tsinghua.edu.cn/apache/spark/ name: sparkIn the previous version, the mount points declared in the data set were mounted to the virtual file system only when the data set was created. Once the data set is created, even if the mount point is modified, it will not take effect. If you want to modify it, you can only delete and rebuild, which has a lot of inconvenience in actual use. In Fluid v0.6, we have added the function of dynamically increasing and decreasing data set mount points. Once the mount point of the dataset is modified, the dataset will temporarily enter the Updating state. The data set will not return to the Bound state until the mounting/unmounting is complete.
New and enhanced cache engine
1. New data cache engine implementation
The default distributed cache Runtime used by Fluid is Alluxio Runtime. In order to meet the needs of users in different environments for the cache system, Fluid has designed the distributed cache Runtime access framework into a pluggable architecture in the previous version. In Fluid v0.6, community contributors from Tencent Cloud developed GooseFSRuntime based on this framework, and added an execution engine implementation that supports Fluid Dataset data management and caching.
2. Data caching engine high availability runtime
The cache engines Alluxio, JindoFS, and GooseFS currently supported by Fluid are all distributed file systems, and the Alluxio, GooseFS, and JindoFS Cache modes are all Master-Worker architectures. In a distributed file system, high availability includes two aspects: one is the availability of the entire file system, and the other is the integrity and consistency of data. As a global metadata management component, Master ensures the high availability of the file system through Master High-Availability; uses the Raft algorithm to achieve master selection, state machine synchronization and other operations to ensure the integrity and consistency of logs and metadata. In Fluid v0.6, community contributors from Tencent Cloud designed to implement high-availability runtime functions. Users only need to specify the desired number of masters. If the number is an odd number greater than 1, the high-availability mode will be automatically used.
summary
Fluid v0.6 mainly solves and satisfies the problems and needs of community users in the actual production environment. In terms of scheduling, it supports data-aware Pod scheduling to meet the needs of coordinated scheduling of data and applications in multiple scenarios. In terms of data set operations, related functions have been further enriched, supporting automatic elastic expansion and contraction of data sets, and dynamic update of mount points. In terms of cache engine, the new cache engine has been added and enhanced, which supports the high availability of the cache engine and adds a new public cloud cache engine.
learn more:
https://fluid-cloudnative.github.io/
Thanks
The development of Fluid v0.6 has received contributions from many units. Here are the names of the developers of several important functions provided by this version. Thanks for their contributions!
● Yang Li (Beijing Qihoo Technology Co., Ltd.) data set mount point dynamic update function;
● Qiu Lingwei (China Telecom) data cache-aware Pod scheduling optimization;
● Xie Yuandong (Tencent Cloud) cache engine high availability, GooseFSRuntime support;
About the Author
Dr. Gu Rong, associate researcher of the Department of Computer Science of Nanjing University, Chairman of Fluid Open Source Community, member of Alluxio open source project PMC, research direction big data processing system, has published more than 30 papers in cutting-edge journal conferences such as TPDS, ICDE, JPDC, IPDPS, ICPP, etc. , Presided over a number of general projects/youth projects of the National Natural Science Foundation of China, and special funded projects of the China Postdoctoral Science Foundation. The research results have been applied to Alibaba, Baidu, ByteDance, Sinopec, Huatai Securities and other companies and open source projects Apache Spark, Alluxio, won the 2018 Jiangsu Science and Technology First Prize, the 2019 Jiangsu Computer Society Youth Science and Technology Award, and Tencent Cloud's Most Valuable Expert Award. Secretary-General of the Big Data Committee of Jiangsu Computer Society.
Copyright statement: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright, and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。