Industry Pioneers Talk About Flink's Future - FFA 2021 Roundtable (Beijing)

Abstract: On January 8, Flink Forward Asia 2021 invited several leaders of real-time computing from top domestic and foreign technology companies to hold two roundtable sharing sessions. This article brings you the wonderful content shared by the roundtable titled "Industry Pioneers Talk about Flink's Future" in Beijing. This roundtable mainly discussed three topics:

1. How do you see the future of Flink and real-time computing?
2. How do you view the relationship between Flink and other open source projects?
3. How do you view the relationship between the enterprise and the open source community?

FFA 2021 Live Replay & Presentation PDF Download

1. How to view the future of Flink and real-time computing

Can it be considered that Flink has matured in real-time computing? What does the future of real-time computing look like in the eyes of the distinguished guests? Before realizing such a future, what areas and problems does Flink need to explore?

Wang Jiasheng: The function is becoming mature, and the application field needs to be expanded

I personally think that there are two more important signs of maturity: functions tend to be perfect and large-scale applications in core scenarios. I think Flink has matured at the functional level now, including support for real-time computing and streaming batch integration. But in terms of applications, Flink still needs to promote more businesses. Expect Flink to play a greater role in more fields.

Dong Tingting: Streaming computing is becoming more mature, and there is still a lot of room for exploration in stream-batch integration

In the field of streaming computing, Flink is already the de facto standard in the industry, the first choice for companies to select streaming computing models, and is becoming more and more mature in terms of stability and ease of use. As a stream-batch integration engine, Flink still has a lot of room for exploration in batch processing, such as dynamic resource scheduling, speculative execution and other capabilities, as well as application scenarios such as lake-warehouse integration and OLAP. In this regard, the community and various companies are also actively promoting and exploring, and it cannot be said to be mature.

Ju Dasheng: Flink is a trend, and the future will be brighter

From the perspective of data processing, Flink has reached a trend at this stage, but it is not yet mature. Measuring maturity depends on two aspects: from the actual usage in business application scenarios, the current trend of Flink in streaming processing has been achieved, batch processing is still dominated by Spark, and data lakes and incremental production have just started , so from the application point of view, Flink's mission is still a long way to go; in terms of technical capabilities, Flink still has room for improvement in batch processing, large-state job execution, and disaster recovery. It is expected that Flink will play a greater role in the field of data processing and grow more capabilities and scenarios.

Zhang Guanghui: The streaming computing scene is relatively mature, and the technical level is further improved

Flink is relatively mature in streaming computing scenarios, and has been implemented on a large scale in real-time data warehouses, real-time ETL, and real-time machine learning. coverage has been wider. However, at the technical level, there are still some areas that need to be further explored in the future, such as the separation of storage and computing between state and computing, the ability of elastic computing to cope with burst traffic, and better fault tolerance.

Wang Feng: Stream computing must go to the extreme, real-time computing is not just stream computing

The overall trend of big data is to evolve towards real-time, and Flink plays a pioneering role in it. Flink has matured in terms of stream processing technology, but it has not yet reached the extreme. For example, how does Flink dynamically adapt to elastic expansion and contraction under the challenges of cloud native, how to achieve a complete separation of storage and computing, and how to improve fault tolerance to achieve zero downtime. I think in terms of stream computing, Flink should evolve towards the extreme next.

The big concept of real-time computing does not equal stream computing. When the data changes, it senses the change immediately and processes it according to the predetermined logic. This is what stream computing does. At the same time, there are other forms of real-time computing, such as millisecond-level and second-level processing for a batch of data. Analysis, which is still the realm of real-time computing. Flink still has a lot of room for development in terms of efficient analysis of stock data. Relying on its stream-batch integration capability and efficient computing engine, it can realize real-time data in larger scenarios.

2. How to view the relationship between Flink and other open source projects

At present, there are many successful cases of combining Flink with other ecological projects in the industry. At the same time, Flink also faces competition with other open source projects in many aspects. What do you think about the relationship between Flink and these ecological projects? How should Flink be positioned and differentiated in the entire open source big data ecosystem?

Wang Feng: Keeping Open, Innovative Development, and

Flink needs to remain open, which is the basic idea of an open source system. We need to work well with other open source projects to form an ecosystem. At present, Flink has a good connection with the upstream and downstream systems. It can connect to different databases and data warehouses through the connector. The deployment is also compatible with the ecology of Hadoop and Kubernetes. We need to keep going in the future.

There may be some overlap between Flink and some ecological projects, and this competition is benign in my opinion. What users need is a complete and convenient experience, and it may be difficult to distinguish the boundaries of various engine components. The current trend seen, each ecological project is expanding its own boundaries, taking a step forward based on its own core advantages, making it more convenient for users to use, this is a healthy competition. The same is true for Flink. Based on the core advantages of a pure streaming execution engine, it can build an integrated stream-batch analysis capability, which is compatible with some offline scenarios; based on Flink SQL, it can perform short queries, OLAP analysis, analysis of limited data sets, and query acceleration. ;Flink will also have its own innovation in the storage data format of stream-batch integration in the future, realizing the combination of stream-batch computing and stream-batch storage, realizing the concept of streaming data warehouse, and realizing a one-stop big data analysis experience.

Therefore, I think the major ecosystems should not only maintain openness, be able to integrate and connect with each other, but also have their own innovative development to give users a better experience.

Zhang Guanghui: Cooperation to achieve one plus one greater than or equal to two

From the current situation, Flink and other ecological projects are more cooperative. For example, to build a real-time data warehouse, Kafka is required upstream, and Flink is used in the middle to import the calculation results into Doris for real-time analysis. I think the cooperation between Flink and other ecological projects can achieve a state where one plus one is greater than or equal to two.

Ju Dasheng: Correctly view the competition and cooperation relationship, give full play to the advantages to create value

Cooperation and competition are frequently encountered topics in the open source community. It is very important for us to correctly view the relationship between cooperation and competition, analyze our own strengths and weaknesses, make better use of our strengths, and provide users with more capabilities. Specific to this problem, I think there are currently three parts in the industry: one is the lightweight event processing capability on MQ; the other is data lakes similar to Kafka and Hudi. I think these are cooperative relationships. Through real-time update capabilities, data warehouses can achieve more incremental production capabilities, not just batch or streaming data warehouses; the third is OLAP scenarios, in this regard I think it is indeed a competition and cooperation relationship. The so-called competition means that in the production and application of these two projects, when one of them has a strong capability, it will surely occupy the other's market, which is inevitable. And no matter which of the two is doing better and better, it will definitely provide users with more value, which is from the perspective of cooperation. For example, we propose real-time data warehouses and use Flink + Doris. They are cooperative, but to a certain extent, they are also competitive.

Regarding how to maintain differentiated competition, I think Flink can see where it is weak and improve it. For example, the status of Flink is transparent to the user, and the end user cannot access the status data and cannot analyze it. In this regard, it can be done better.

Dong Tingting: maintain ecological integration and enhance their competitiveness

Regarding cooperation, Flink is a computing engine that integrates streams and batches. If you want to realize the flow-batch integration of the whole link, in addition to the flow-batch integration of the computing engine, you must also rely on the flow-batch integration of the storage engine. Therefore, Flink can maintain a good cooperative relationship with storage engines such as Prevaga and Hudi, and integrate more with the entire ecosystem, allowing users to have more choices, and can also cooperate well with other engines to achieve a true full-link flow Batch integration, lake and warehouse integration construction.

Regarding competition, in the field of computing engines, there is indeed some competition between communities. Flink needs to improve its competitiveness, such as whether it can compete with Spark in terms of batch engine capabilities, and whether it can be substitutable.

Wang Jiasheng: Focus on computing and maintain a cooperative relationship with storage systems

With the development of technology in the field of big data, the separation of storage and computing is more in line with future trends. On the one hand, the storage layer and the computing layer iterate separately, with faster speed and more advantages; on the other hand, regardless of the storage layer or the computing layer, it is difficult for a single engine to support all scenarios for a long time, and it still needs to be covered by multiple engines. In business scenarios, the separated state can meet the needs of various scenarios more. Therefore, Flink should focus on the computing level, and it needs to have a good cooperative relationship with related projects at the storage level to provide a better user experience for the business.

Projects such as Clickhouse and Doris are also exploring streaming computing. The biggest advantage of these projects is query efficiency. The advantage of their exploration of streaming computing is to reduce the complexity of user usage. The weakness of these projects compared to Flink is that Flink is more advanced in real-time computing and has better support for the real-time computing field. If Flink is further improved in terms of user experience, access thresholds, etc., coupled with its powerful functions, it can still maintain a very strong competitiveness, covering more application scenarios, connecting more systems, and supporting more business scenarios. In the ecology of separation of storage and computing, it occupies an important position at the computing level and maintains its advantages for long-term development.

3. How to view the relationship between enterprises and open source communities

What are the advantages of using and contributing to open source projects, and what challenges have you encountered along the way? How do you view the relationship between practical innovation within an enterprise and the open source community? What are the Flink explorations that the teams of the distinguished guests are working on, what plans are there next, and what innovative technologies do they plan to contribute to the community?

Wang Jiasheng: actively embrace open source, one warehouse to explore the lake, stream batch one

Xiaomi's attitude towards open source is very open. We actively embrace open source and believe that open source has great advantages. On the one hand, with the power of open source, we can support very complex business scenarios and meet the complex needs of various businesses with less human input; on the other hand, by participating in open source, we can work with outstanding engineers in the industry. Carry out technical exchanges and improve their technical capabilities.

Our current explorations in Flink: First, combine Flink + Iceberg to do the integration of lakes and warehouses, hoping to provide the business with a near real-time data warehouse development and use experience; second, the attempt to integrate streaming and batching, through Flink to integrate the company's internal real-time The technology stack for integration and offline integration is unified. Further exploration will be made in other areas in the future. The challenges we encountered mainly come from the project usage threshold and development operation and maintenance experience. Open source projects have given us strong core capabilities. In the actual business application process, it is still a challenge to solve complex problems and reduce the threshold for users to use.

Regarding the relationship between internal practices and the open source community, I think it is necessary for each company to adapt some internal features based on the open source version according to its own special scenarios and needs. In view of these characteristics, the overall principle is to try not to erode the core code, and to avoid being separated from the community and causing difficulty in operation and maintenance. In terms of iterative development of other functions, we insist on participating in discussions in the community first, absorbing suggestions from outstanding engineers in the industry, ensuring that our design is reasonable and correct, and striving to be merged into the community.

Dong Tingting: Open source has many advantages, version follow-up is a challenge

There are many advantages of open source projects. For example, we can communicate with the community, absorb good ideas and design suggestions when encountering problems, and reduce the cost of trial and error; we can also enjoy some dividends brought by community code, Learn from the community's proven solutions and avoid reinventing the wheel.

Over the past period of time, Kuaishou has also cooperated with the community in Flink SQL, including follow-up, participation in some functions of the community, development of issues, and also contributed to our internal self-developed functions such as progressive windows. In terms of the engine, we have made hot update models, current limiting strategies and other functions that are well received in our internal business usage scenarios, and are also discussing with the community to prepare to contribute these functions to the community.

We had some challenges in keeping up with the latest release from the community. For example, some internal self-developed functions are not very general, and are suitable for some scenarios within our company. For the new version released by the community, we need to cherrypick these functions to the community version. There will be additional labor costs in terms of Cherrypick code and promotion of business upgrades. So we basically follow up with the latest version from the community every other year, merging internal features together.

Ju Dasheng: Open Source Projects Accelerate the Development and Application of

I think open source projects can very well accelerate the development and application of a technology or field. Specifically, a company can quickly start exploring in this area with the help of open source projects. In addition, open source projects can concentrate problems in related fields, let everyone participate in solving problems in related fields, promote rapid iteration and resolution of problems in related fields, and promote the development of related fields. The advantages of these aspects are very obvious.

Meituan's Flink planning is basically divided into three aspects: in terms of real-time data warehouses, it is expected to build a real-time data warehouse development scenario through the Flink + Doris link; in terms of incremental data warehouses, through Flink + Hudi, near real-time data can be achieved Output effect; in offline scenarios, it is expected to unify the engine to Flink. We hope that the business side can choose different scenarios based on cost and timeliness, keep the language and the underlying engine consistent, and provide the business side with such convenience.

The challenges encountered in the process of using open source projects are two-fold: First, we found that open source projects generally do not pay much attention to operability. Operability needs to be combined with the company's internal ecological technology. The open source community does not know the company's internal operation and maintenance environment, and pays less attention to this aspect. This is also the first difficulty that most companies need to solve when using open source projects. Second, when working on open source projects in depth, if the development of the community is relatively slow, the company may not be able to wait and need to do a lot of iterations of features and even large versions. This is a challenge for the company, and in turn, a challenge for the speed at which the community can grow.

Next, we may try to integrate streaming and batching in the real-time update of Flink, and hope that the solutions and ideas of these attempts can be shared with the community.

Zhang Guanghui: Enterprises and communities complement each other and promote each other

Byte has done a lot of technical exploration for Flink. In terms of SQL, it is more about functional exploration, such as delayed join, increased aggregation indicators, recovery from checkpoint, and various internal connectors. In terms of state checkpoint, we are more about performance exploration, such as small file aggregation, state user can check, and reducing the performance loss of RocksDB State Backend serialization and deserialization through cache. On the runtime side, we are more about improving stability, such as blacklist mechanism, single point of failure recovery capability, etc.

I think the advantage of using open source projects is that we can implement a better set of technologies in internal business scenarios with less human input and shorter time. The biggest challenge we encountered was version upgrades. The upgrade iteration of the community version is still relatively fast. We have accumulated a lot of features internally and it is too late to contribute to the community. The cost of each version upgrade and business migration is very large.

I think the internal practice and the open source community as a whole are a mutually reinforcing and mutually reinforcing relationship. Specifically, Byte has some tens of thousands of concurrent large jobs, and each restart takes about ten minutes, which is intolerable for our business. We communicated these issues with the community, and the community classmates also actively connected with us, and soon reached a series of solutions. These issues were resolved in subsequent versions, and the restart speed of the homework was increased by 2-3 times.

Future planning: In real-time computing scenarios, we will make some explorations in improving elastic computing capabilities, separating state storage and computing, and improving fault tolerance. In terms of flow-batch integration, we will push more businesses to release the advantages of Flink and liberate and improve business productivity. In addition, we have some exploration and implementation in short query, and we will return some scheduling performance and SQL performance optimization to the community in the future.

Wang Feng: Advocating the combination of open source technology system and enterprise business and talent training

"Open source" is a very hot topic. The national plan has proposed that China should be independent and controllable in basic core technologies, open source and open. It is hoped that domestic enterprises can form a synergy through open source and improve the country's software infrastructure. Alibaba's embrace of open source has become more and more active in recent years, and an open source committee has also been established. The Apache Flink project is also a typical representative. It is believed that companies in the industry can feel Alibaba's support and investment in the Flink open source project. The team I am currently in has dozens of engineers and core technicians working full-time on the Flink open source project.

Dong Tingting and Zhang Guanghui also mentioned just now that the streaming media industry where Kuaishou and Byte are located has a very large amount of business and has encountered many technical challenges. They also hope to contribute the problems solved in this extreme challenge to the community, which may be limited by The community version was released too quickly to integrate better. At this point, our experience and suggestion is that we hope that the leading Internet companies in China can invest more in the construction of open source communities, so that the company's engineers have more time to invest in the construction of open source communities. The more people who invest, the more opportunities to generate Committers and PMC Members, the more opportunities to speak in the community, and the more opportunities to push their own things into the community, the greater the influence on the community, the formation of A virtuous circle. I also hope that the Apache Flink community can attract more companies' contributors to participate and become Committers, so that many problems just now can be solved naturally in the open source community.

There is a very close relationship between the internal technical practice of the enterprise and the open source community. We believe that the contributors and core developers of the open source community all work in a technology-driven company. Many ideas and needs in the open source community actually come from various companies and industries. Therefore, the technical practices of the internal business are the source of the open source community. In addition, the version released by the open source community needs to be practiced in various enterprises and industries, which is also a great advantage of open source projects. Open source projects are free, and everyone is willing to use them to expand the market at a very low cost. After so many users, industries, companies, and scenarios use it, it will provide rich feedback to the community. The open source community can quickly iterate based on this feedback, and then go to various companies to practice. This is a good mechanism for open source projects to develop rapidly and be recognized by everyone. This also includes technological innovation. The open source community is a link. Here, you can see how each company uses the project, what are the problems of each company, and your innovative ideas can also be used by other companies. A very good environment for nurturing technological innovation. I think this is a very good model for enterprises to participate in the open source community, based on the open source and open technology system, combined with their own business and talent training.

FFA 2021 Live Replay & Presentation PDF Download

---

For more technical issues related to Flink, you can scan the code to join the community DingTalk exchange group
Get the latest technical articles and community dynamics for the first time, please pay attention to the public number~