The world's largest open source software foundation, the Apache Software Foundation (hereinafter referred to as Apache), announced on official channels Apache DolphinScheduler to become the top Apache project on April 9, 2021 Beijing time. This is led by Chinese people and contributed to Apache's big data workflow scheduling field.
DolphinScheduler™ already used by 400+ companies such as China Unicom, IDG, IBM, JD Logistics, Lenovo, New Oriental, Nokia, 360, SF Express, and Tencent, as a distributed visualized big data workflow scheduling system.
DolphinScheduler was donated by Analysys to Apache on August 29, 2019 and started its incubation. After that, under the guidance of the mentor, the members of the incubator management committee will conduct tutoring and incubation. On February 19, 2021, the Apache incubator will vote for the graduation proposal with 8 votes in support. On March 18, the Apache board of directors passed the DolphinScheduler graduation resolution, ending the 18-month incubation period, and officially determined that DolphinScheduler became the top Apache project.
On April 9, 2021, Sally Khudairi, Vice President of Apache Marketing, issued an official announcement on the official accounts of the Apache Software Foundation.
About Apache DolphinScheduler
Apache DolphinScheduler is a distributed and extensible new-generation workflow scheduling platform with a powerful visual DAG interface. Since the establishment of the project in 17 years, the mission has been determined-"Resolve the intricate dependencies between big data tasks and enable The entire data processing process is intuitively visible", and no code is required to configure the workflow from now on!
DolphinScheduler architecture
As a powerful distributed big data workflow scheduling platform with a directed acyclic graph (DAG) visual interface, Dolphin Scheduler solves complex task dependencies and simplifies the work of data task scheduling. It uses out of the box , easy to expand way to connect many big data ecological components to the 100,000-level data task scheduling system.
Apache DolphinScheduler is a data scheduling platform recognized by the world's top architects and the community. It leaves complexity to itself and ease of use to users. The overall features are as follows:
- Cloud native design: supports multi-cloud, multi-data center cross-terminal scheduling, and also supports K8s Docker deployment and expansion. Performance can grow linearly. In user testing, it has supported up to 100,000 parallel task control;
- High availability: The decentralized multi-Master/Worker architecture can automatically balance tasks and automatically high availability, ensuring that tasks can complete the overall scheduling with integrity in the event of any node crash;
- User-friendly interface: visual DAG diagram, including convenient functions such as subtasks, conditional scheduling, script management, multi-tenancy, etc., and has the ability to separate running task instances from task templates, so that your platform maintainers and data scientists have a convenient Easy-to-use development and management platform;
- Support multiple data scenarios: support streaming data processing, batch data processing, pause, reply, multi-tenant, etc. It can be easily and directly called for platforms such as spark, hive, MR, flink, clickhouse, etc.
The fact that DolphinScheduler can harvest so many users in just one and a half years of incubation time is inseparable from the Slogan it has always advocated:
Yes, the call is right, I can sleep at ease in the middle of the night! Eventually Apache Dolphin Scheduler has gained a large number of users, such as China Unicom, IBM China, JD Logistics, Lenovo, New Oriental, SF Express, etc. More than 400 users are using it online:
Apache DolphinScheduler has started a new starting point!
At the beginning of the design, DolphinScheduler adheres to the core concept of simplicity, easy-to-use and easy-to-maintain products. It uses high availability and easy expansion as the architecture base. After many polishings, it has been affirmed by the user community, and then it was officially officially launched in March 2019. It is open sourced to the outside world and successfully entered the Apache incubator on August 29, becoming the first domestic Apache big data scheduling project. It has repeatedly logged on the github monthly trend list and caused heated discussions on Hacker News. In 2020, DolphinScheduler has also won many praises. InfoQ China's annual list of technical forces rated it as the "Top Ten Open Source New and New Projects" and "Most Popular Open Source New and New Project", and was awarded the "Best Technical Team" of Open Source China. . The DolphinScheduler community already has a huge 4000+ users. According to incomplete statistics, as of October 2020, 400+ companies have used DolphinScheduler in production environments. DolphinScheduler continues to shine in various fields!
Development history of DolphinScheduler
In the process of Apache incubation, I learned and practiced the Apache Way, and truly realized the essence of Apache culture "Comuunity Over Code". The community is based on code, but it is bigger than code. You need to think about how to collaborate, how to communicate, and how to make the community healthy. Continue to move forward. It is this constant thinking that makes DolphinScheduler not encounter any challenges in ending the incubation voting, which can be said to be unexpectedly smooth.
Graduating to become a top-level project has witnessed the day and night efforts of many contributors and users from the Apache DolphinScheduler community in the past year and a half. Since entering the Apache incubator, the DolphinScheduler community has joined 14 Commiters and 2 PMCs. They are also in the community. 8 Apache Releases have been released under the joint efforts of .Thanks to all the mentors, contributors, users and Apache incubators who have given their hard work and countless help.
Becoming a top-level project is just a new starting point for Apache DolphinScheduler. There are more challenges waiting for the support of friends and users in the community. We hope that in the future we can let China’s excellent open source projects go to the world and let the world know that there are top-notch domestic projects. Finally, I would like to thank you all for your help and blessings for the Apache DolphinScheduler project. We would also like to thank you all, and hope that you will continue to support the Apache DolphinScheduler project to go global!
new generation of big data scheduling-DolphinScheduler related resources
- DolphinScheduler official website: https://dolphinscheduler.apache.org
- DolphinScheduler github：https://github.com/apache/incubator-dolphinscheduler
- DolphinScheduler Twitter: https://twitter.com/dolphinschedule
- DolphinScheduler Slack discussion channel: https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
- DolphinScheduler WeChat Official Account: Dolphin Scheduling
- Add WeChat account to WeChat user group: easyworkflow
**粗体** _斜体_ [链接](http://example.com) `代码` - 列表 > 引用。你还可以使用@来通知其他用户。