Hu Xiaoman: MindSpore Open Source Operation and Governance | DEV. Together 2021 China Developer Ecological Summit

Content source: On June 5, 2021, the 2021 China Developer Ecological Summit hosted by SegmentFault came to a successful conclusion. At the meeting, Hu Xiaoman, Director of Operations at Huawei MindSpore, delivered a speech on "MindSpore Open Source Operation and Governance".

Sharing guests: Hu Xiaoman, Huawei MindSpore Operations Director

shorthand compilation and release: SegmentFault Editorial Department

Today I will share with you the open source operation and governance of MindSpore. Before I talk, I will introduce myself to everyone. My name is Hu Xiaoman. I graduated in 15 years. After graduation, I have been working as an algorithm engineer. I have been an algorithm engineer for four years and wrote code for four years. I went to Baidu in 19 and worked as a deep learning evangelist. I joined Huawei last year and was responsible for the entire MindSpore. Open source operations of the community. In fact, when I received the invitation to the conference, I was very surprised and very happy, because when I switched from algorithm to operation, I was particularly worried about whether I would be unemployed (laughs). Because the circle is too small, many people don't pay special attention to developer operations. In fact, open source operations can be regarded as a new type of work. Unlike R&D or algorithm engineers who do graphics and NLP, it has a very clear purpose and a very clear career plan. It is very convenient to find any such job on the market. So today I will share with you what exactly is open source operation and governance, and what are we doing this year?

What is open source operations and governance?

What exactly is open source operations? I have summarized a few points.

We believe that the core point of open source operations is to connect people in the circle with technology, and to continuously expand the influence of the circle in innovative ways, so that outsiders can also know what your technical product is. Many people only pay attention to the former, and the content they create is only for the technical people in the circle, but if you want to become popular, you must let people who don't want to read your content can read it. We call it "out of the circle." But the difficulty lies in the fact that many operating methods are very homogenized. A common misunderstanding about operations is that they are engaged in new media, or whether they are advertising in Moments, thinking they are editors, this is actually A misunderstanding. We found that operating students with technical backgrounds on the market are very rare and very difficult to recruit. What we want to operate is a technical framework, so a technical background is indispensable for us. Our current entire team basically has a technical background.

The overall content of open source operations, I divide it into four parts: first part is product operation , including our common content, channels, communities, second part is community governance , including TSC, SIG, WG, etc.; third block is open source cooperation , including functional cooperation and application cooperation, etc.; last block is our most common infrastructure , including CI Kanban system, and digital operation to help you better To understand how the operation effect is.

In summary, we currently have several principles for open source operations. First, we believe that open source operations must understand the technology . Many of the operations teams I have worked with did not understand technology. So in normal work, it is difficult for you to have in-depth communication with him. Regarding the framework or the technical ontology, which appropriate method should we take to let developers better understand it. After coming to Huawei, we require everyone in our group to understand technology. If you don’t, then learn it. Some students are studying liberal arts, we want him to learn Shell script to learn Python, and then slowly learn MindSpore.

Second, marketing and commercial atmosphere should not be too strong . Many people will not be able to distinguish operations, markets, and business BDs. They would say that these three seem to be interchangeable, isn't the operation also used for sales? Isn’t marketing also advertising? It seems that there is no difference. But in fact, there is an essential difference between operations and marketing and marketing staff. We are not pure PR, not pure advertising.

The third piece is our experience, in all the operating blocks, we think we should dare to try and make mistakes. We currently do not have a particularly complete or standardized path for everyone to follow. For many ways, we are trying. After trying it, let’s look at its effect. If the effect is ok, we will summarize the better experience. If it is not good, we will continue to iterate. Continue to iterate, repeat the rounds and then sum up the experience, the more practice, the road will come out.

Let me briefly talk about what MindSpore is. It is a full-scenario AI computing framework open sourced by Huawei on March 28 last year. Some people may find it very confusing. Now that there are TensorFlow and PyTorch on the market, why do we need to build a deep learning framework? Where is its core advantage? Why do you let developers choose your framework and replace the original one?

In fact, MindSpore has several very core points. You can see the "automatic parallel" and "second-order optimization" on the right. These two points allow you to train the model very efficiently. For example, if you originally need a week or a day to train the model, then under other conditions unchanged, using MindSpore, you can greatly shorten the model training time, which is a great advantage for many companies in actual production. The second is that the conversion of dynamic and static images is very convenient and suitable for developers to switch. The third is full-scene deployment collaboration, which simplifies the deployment process for developers and has very good ease of use. The next few points will not be discussed in detail. Today, I will mainly talk about open source operations. Here is just to introduce you to what MindSpore is.

We have been operating the open source community for a year. Overall, as of the end of May 2020, MindSpore has accumulated more than 350,000 downloads, and the overall number of PRs is 26,000. At present, the community has launched more than 120 models and more than 2,000. Online applications. It doesn't matter if you haven't used it before. If you use a Huawei mobile phone, the Huawei mobile phone uses the model application trained by MindSpore Lite, which is called more than 300 million times a day.

Core content of open source operation and governance

Next enter the topic. What I want to share with you today is to use model thinking to do open source operations.

I used to do algorithms. We have to train various models every day. Now I find that this idea can be used in operations. Think about it, everyone. When we are working on an algorithm, we get a project. The first thing we pay attention to is how to analyze the needs of the project and how to meet the business side. Then after analyzing the requirements, we have to clean the data in a large amount, obtain high-quality data, and then perform model training and model evaluation, and then adjust the model according to the effect. If the target is reached, we can go online and deploy the model.

Putting this idea into operations is the same. Operations also need to first understand the project to be done now and what goals to achieve, and then decompose the task. For example, if we want to open source operations of MindSpore, what is our goal for the first year, what is our goal for the second year, and what we want to achieve after the third year, we need to think clearly. Therefore, we must first determine the goals for the first three years, especially for the first year. After the goal for the first year is determined, we need to break down the goal. For example, the total number of downloads we want to achieve for MindSpore in the first year is 100,000. How did this number come out? In fact, the number of 100,000 is not a very clear goal. It does not mean that other companies have 100,000 downloads in the first year. We also have to be 100,000. We just think that to reach this number, we can prove that we have an upward trend, and we have a basis that is worth continuing to move forward. There is a correlation between the two, not causality.

So after decomposing the indicators, all the operating activities we are doing must meet the minimum MVP principle. What does it mean? That is, any activity we do will meet the minimum available product principle. For example, TinyMS, a technical project developed by our team, is an advanced API tool based on MindSpore. This small tool is designed to make it easier for everyone, especially Xiaobai, to learn MindSpore and get started with AI. When we were working on this tool, although everyone in our team had a technical background, some students were not pure algorithms. Then we must figure out how to raise the smallest usable product form of this product, and we need to design a solution to user pain points. At the same time, the function cannot be overly complex technical architecture. So I designed the overall technical architecture, and made the core functions with the research and development in the group. A version was released in three months. After the release, I watched the developer’s response and then continued to adjust. The next stage is the second When each version is iterated, make corresponding adjustments. Therefore, after the minimum MVP goes online, we need to make some assessments of its effects. Regardless of whether we are doing technical projects or doing any courses, we will see what core indicators are transformed by the activity, optimize after evaluation, and iterate repeatedly, knowing that we have summed up a reusable experience. Methodology, when you do new products/tasks in the future, you can do it according to the process, which saves a lot of manpower and ensures the stability of product quality.

We can see that the algorithm model and the operating model have many similarities. is that both can be used as a project system, and must meet the minimum MVP principle to put the project online as soon as possible, instead of spending a month or two to do a class. This is a waste of time and the market will not wait. you. The second is that all need to iterate repeatedly. You can iterate over and over again, find the problem, and keep optimizing it, so that your effect can continue to improve. The third is that whether it is algorithms or operations, it is necessary to achieve differentiated competition. For example, in the recommendation algorithm, Taobao's recommendation, Jingdong's recommendation, and Pinduoduo's recommendation must be different in mechanism. If the three are the same, there will be a certain share of the user's choice in the loss. Especially for operations, if you don’t have differentiated recommendations, today users see other manufacturers have done courses, they can choose their courses, and the next day they see the courses we do, then why should users choose us? Where is the difference between us and them? Is our teacher stronger and the curriculum design more interesting? Or is the service experience we operate better, so that all users can answer some questions after class and get complete learning feedback? This is a point that we need to think about frequently when doing projects.

But they have several differences. The first is that the algorithm model in the company requires more attention to a large amount of data. Because no matter how the network structure is optimized, based on experience, more often it is better to make the data good enough and add enough high-quality data to train the model. But operations are more concerned with actual data. Be wary of the false prosperity of data indicators. Although it can reflect the results of your work to a certain extent, it will also deceive the points you need to improve and improve. Of course there will be people who disagree, thinking that the data can reflect everything. We don’t argue about this argument. What I want to express is that the category of open source operations is relatively new. You can’t find data that fits your current operational needs, help you do preliminary research, and give you some guidance and suggestions before you do the project. . The survey data of others and your existing statistical indicators of prosperity data may not give you the true and correct indicators. The second point is that the algorithm model is often optimized and iterated repeatedly, and then goes online after reaching a certain threshold, but operation is not. The most important thing for operation is to quickly produce the content you expect. You have to set a baseline for yourself, that is, set a baseline version, rather than saying that I have completed all the functions of the product before it can go online. When you finish it and then go online, other competing products may have already come out and won't wait for you at all. The third is that the algorithm model will pay very little attention to outliers or individuals, especially our attention to this kind of divorce is not particularly large (in most scenarios). But I think it’s still very important for the individual to focus on the operation, because you only know the developers, especially one-on-one or chat with enough developers, to know what the core pain points of the product they are really concerned about are. After you talk with him, you can do some refined operations and find their core demands. Constantly assume that you are a user, and truly experience the user scenario, and chat with real users, in order to dig out the essence of the demand. This is not only a very important point for operations, but also for products.

The entire system structure of open source operations can be divided into these parts.

first part of 160f7a2d0c60fc is brand marketing. brand marketing, we will do some large-scale activities. The purpose of the activity is only to enhance the influence of the brand, without direct conversion of key indicators. Many people will ask you what kind of benefits you have brought from doing this kind of large-scale activities, and those who do the market may not explain exactly what kind of benefits it brings. You can say how much attention and broadcast this conference or the entire network has, but the boss will ask you, what benefits does it bring? We finally came to the conclusion that for brand marketing, we do not convert key direct indicators. You see, when we do large-scale meetings and large-scale events, can we really improve our core indicators like the number of broadcasts? Not necessarily, but making more users aware is also a very important part.

second block of 160f7a2d0c6114 is the core content of open source operations. This piece of content is very much, we have a variety of different formats, including articles, graphics, videos, podcasts and so on. The forms are very diverse, and we must achieve high-frequency output. For each type of activity or content, we will set specific indicators to improve. For example, for courses, we want to improve the core developers, so the core developers, we will not put it on your podcast or make some short videos that are better for developers as his indicators to improve .

third block of 160f7a2d0c612b is college expansion. University expansion is to solve the source problem of users, and college expansion is to develop the base of developers, collect student suggestions, and improve the overall framework or the applicability of the overall product.

last piece of 160f7a2d0c613f is the enterprise landing. This year we will work with Huawei’s AI Computing Center ascendant innovation center to do corporate activities in the core area, provide computing power and empower companies, expand some corporate landing projects, and increase market share.

Brand Marketing

Let me tell you what brand marketing we do. For example, at the beginning of May this year, in CCTV-2 "Economy Half an Hour", I introduced to you the achievements of MindSpore this year. The second piece is the anniversary of MindSpore. Because MindSpore was open sourced on March 28th last year, we held an anniversary event on March 28th this year. We invited members of the TSC committee, more than 40 technical experts and core developers around the world, and corporate users to record their wishes. Video and technology sharing. Finally, the MindSpore Tucao Conference. Have you seen the Tucao Conference, why are we doing the Tucao Conference? Because many developers have some suggestions for our products, but they have no way or channels to make some very intuitive complaints. They basically raise issues or go to QQ groups/forums to ask questions, so we provide developers with such a channel. Let everyone focus on complaining. This can not only shorten the distance between the developer and us, but also let our core developers know where the product is likely to make the developer have a bad experience. Actually, the effect is very good. In addition, on the left are some high-end activities, and on the right are the credible open source community evaluation system certification just passed this year. MindSpore is the first batch and the only AI framework that has passed credible open source evaluation. Its evaluation includes not only the technology, but also the operation of the community, the norms of the community, the governance of the community, etc. It is not easy to score every aspect.

core content

Next, look at the core content. Let me talk about the content of the product first. Our section includes not only self-produced content, but also developer content. If we produce it ourselves, it is equivalent to playing by ourselves with the door closed. We must encourage developers, especially core developers, to contribute content to us. Developer content not only includes ordinary developers, community developers to write articles for us, or he wrote some technical articles after participating in the event, we will also contact some core KOLs to give us some core technical highlights videos.

This is a technical project done by our team and I have just told you about it. In fact, many people have stereotypes about the operation team-can the operation team also do technical projects? Including our company's internals will also find it very strange: Hey, did you find R&D for this project? In fact, it is completely unnecessary. At that time, we wanted a novice user to contact the MindSpore framework. It was difficult for him to get started. He had to be familiar with its API and overall architecture. This was a barrier for pure novice developers. The purpose of TinyMS is to lower the threshold, so that pure white developers have no threshold to learn AI and can learn AI projects from scratch. The overall architecture of TinyMS is very simple. There are many data sets built in the data class module, and many commonly used deep learning models are built in the Model class. For developers, you only need to write a line of code to call the data set. , Model training and model deployment.

Except for technical projects, basically all our courses will be recorded as a series. In the upper left corner is the 21-day training camp, which is a fun practical course we did in October last year; the upper right corner is the two-day training camp after each iteration of the new version is released, so that developers can learn about our iterations most quickly New features; the following is a nanny-level tutorial for TinyMS; on the right is SIG meeting, TSC meeting and so on. For the same series of videos, we all use the same cover, the style is very uniform, and the focus is also very prominent, which is convenient for developers to find videos. The official account is more conventional, so I won’t go into it in detail. The overall requirement is that we hope that our operation team can maintain the same color tone of the main KV when doing the official account, because our main logo is gradient blue, and then uniformly released during peak hours.

For the operation of MindSpore B station, I have just told you what the current content of our B station is. If it is released, it will be done during the peak time. All these videos have now been uploaded to the CCF electronic library. You can watch them not only on station B, but also in the CCF digital library. Currently, we are the only AI computing framework that has a complete video learning courseware on the CCF library.

When it comes to Douyin, many people are actually curious about why we should do Douyin. Because it is generally understood that Douyin is not a community where technicians love to visit, many programmers do not have Douyin on their phones. But we thought that now developers are getting younger and younger. And MindSpore just came out. Many people don't know what MindSpore is, what are its highlights, and where it compares with other frameworks. By convention, we always have a release note when we release new versions, right? There will be a series of specific updates, these updates are too long for developers as new users, do not know which one is the focus. So I wonder if I can explain the highlights of our release with a one-minute video. Key point: Don't take more than one minute, use stories close to the lives of developers to tell the technical features of our iterative update.

For example, the video in the lower left corner, this is the 0.6-beta version. One of the core points of this version was Wide&Deep. At the time, Wide&Deep was trained with ascending 16pcs, which can be trained in 23.6 minutes, which is the highest performance among all current frameworks. If I say such a sentence directly, the insider will understand it, but the people who do not recommend the direction will not understand it, and even the outsider will not understand it. So at that time we set up such a story. The heroine wants to know what his preferences are. You can use MindSpore's Wide&Deep model. It only takes 23.6 minutes to complete the training and make a "guess you like" system. Then I put the label of "love" and made this story into a short video. After it went online, it reached 100,000+ views within 24 hours. I did not do any advertising because Douyin is too expensive and we don’t know the effect. We just rely on the content to attract everyone’s attention to our product highlights.

In addition to Douyin, we also operate a video account and a podcast account. The video number is mainly based on the content of the developer, the theme is relatively changeable, and the format is very interesting. But the podcast account is different. The content of the podcast account focuses on interviews with AI technical experts. We will share the things that have been implemented in the AI industry, which will be different from the video account.

The overall system between communities is set differently from the conventional system.

You can see the picture in the lower right corner, we are divided into SI, ST, GI, GT and so on. It is a split of MindSpore words, with the words Gradient and Jacobi added in the middle. Developers at each level will have a number, and the number is unique. For example, if you participated in the certification of our event today, then we will give you the certification number. Your code name in the community is unique. This code name is different from your GitHub account or other accounts, which is your personal identity. This account represents your identity in the community. After we launched this, more developers are willing to participate or walk into our community, just to get the community's certification number, so that he feels a sense of belonging and presence. In the area of KOL operations, at station B we found the popular science KOL teacher Li Yongle to explain to us in a very simple way what MindSpore can do.

The live broadcast is actually very common, and everyone will do it too. However, we actually have a distinction in the live broadcast. We will do live broadcasts on Douyin and Station B. Station B is more conventional, sharing technical content. It is a purely shared live broadcast, and will not face the audience or always pay attention to the game. To communicate with the scene, it is generally done after the talk is finished, and then to see if there are any bullet questions before answering. Douyin live broadcast requires PK mode to connect with other people. For developers, it is a very new and immersive interaction. I can invite MindSpore technical experts to talk about how the technical features are produced and what problems are encountered during the development process, which will make developers have a deeper interest or understanding of this technical feature. In addition to looking for technical experts to do the live broadcast, we will also invite developers to do the live broadcast. All of our excellent developers need to make live replies on Douyin to share their stories with MindSpore. Similarly, MSG organizers also need to go through a defense. If you become an MSG organizer, what content and experience should be done, and all live broadcasts will be publicly placed on station B.

As for the course, the 21-day training camp is a fun course for junior and intermediate developers. In many courses, especially deep learning courses, most of the first class is handwritten digit recognition, which is "aesthetic fatigue" for many developers. If I want to learn MindSpore, the first lesson is also to learn handwritten digit recognition, I will feel very boring. We wondered whether the first lesson could be practiced with a relatively new curriculum. At that time, there was a hot search on Weibo. A young man in Yunnan accidentally ate poisonous mushrooms and had hallucinations. So we wondered whether we can use MindSpore to identify poisonous mushrooms to help developers or people in daily life detect whether mushrooms are poisonous. We immediately developed the model, found the mushroom data set, and did a case study. Unexpectedly, the response was quite good. After everyone made it, they would take a variety of mushroom pictures, including those of Super Mario, to test. This can bring developers closer to the framework technology itself. In addition, there will be a two-day training camp. Since last year, we will do an iteration and release a new version at the end of each month, and the new version will have new features. In addition to the technical video just mentioned to let people outside the circle understand the new features, people or developers in the circle want to know that these features cannot be used in the actual production process. So we did a two-day training camp to let the technical experts who developed these features talk about what the features did, and share some live broadcasts of the technology.

The following is MindCon Geek Week. MindCon Geek Week was an event done at the end of last year. The cost of the event was very low, and almost no cost was spent, but the final effect was very good. We finally received more than 10,000 D0 level developers from me, and incorporated more than 20 bugfixes. How did we do it? At that time we created an organization MSG. MSG is the abbreviation of MindSpore Study Group, that is, MindSpore Study Group. What we did last year was a regional organization. We will do it in various cities, including Shanghai, Beijing, Shenzhen, Suzhou, Hangzhou, Tianjin, etc. Such regional organizations. Let all city organizers compete: the more bugs people in which city solve, the higher the points will be. We will have a material incentive to get the highest points in the end. Although there are not many incentives, it will very much stimulate the interest of developers. When they finally come to solve the bugfix, they have been attracted by the process of solving the bugfix and participating in the open source process, instead of just participating in the activity for the reward. So even if it didn't get the first place, it would participate in the open source contribution, from 0 to become a contributor to the open source transformation.

Just mentioned a word: MSG. In fact, everyone sees that TensorFlow and PyTorch will have their own regional organizations, and the organization of TensorFlow is called TFUG. We can do our own urbanization organization, MSG is to organize some offline activities in different regions. Last year we mainly did regional organizations. At the end of the year, we did some university-based organizations. This year we have sorted them out more systematically, including regional, university and enterprise types. From July of last year to now, in less than a year, we have done 13 cities in China, 7 cities overseas (of course, all overseas are online), we have done five campus trips, and finally we are doing business In an attempt, we combined with Magnolia Open Source, Yunqi Capital, Julia Community, Graviti and other start-up companies to create a start-up company MSG, to understand their company, especially the open source start-up company, what are we here? which provided. Enterprise business is very profitable for investors, because they can find some excellent start-up open source companies; it is also very helpful for us, we can understand how the market is measuring the open source community. Standards, in order to guide our work, to better understand the work; for participating companies, they can also learn how an excellent open source community should do it. Therefore, it is a trilateral win-win situation. In the future, we will do greater and more exploration. In the second half of the year, we will focus on the Shengteng Innovation Center and conduct corporate activities in the core area to provide computing power and empowerment to enterprises, and continue expand.

In the technical competition, we have different activities for students, developers, and universities. For example, the student-oriented competition is the summer 2021 open source software supply chain lighting plan, allowing everyone to participate in the community to do some specific tasks; for developers to do some TinyMS model recurring competitions. When we were doing it, TinyMS 0.1 version was released on March 31. We released four models for everyone to reproduce. Is the model you made with other frameworks, can you use TinyMS to reproduce it? Do you know how long was the first submission for recurrence? Within 17 hours, developers made the first model reproduction based on TinyMS. Very unexpected and pleasantly surprised! So far, one month after we released, all four models have been submitted. Our competition is set for three months, and the enthusiasm of the developers and the recognition of the product have far exceeded our expectations at the time.

We won’t go into details about community operations. We include QQ groups, WeChat groups, and Slack groups. The total number of communities has reached 20,000+. Many groups are relatively hot at the beginning, but after a period of time, they die or everyone stops talking. This is actually very problematic. If everyone does not speak, we will continue to throw away some technical issues, or some bright things, to stimulate the interest of developers, let everyone continue to discuss in the group. You can also disperse active developers in different groups, so you don't have to all be operated by official staff.

This is the new operation activity we tried last year, the topic operation of the circle of friends. There is a new function in Moments: add #, bring the topic, click to see which Moments of your friends use this topic in your Moments. We provide developers with the MindSpore Lite apk. After this apk is installed on an Android phone, you can use it to take pictures directly to detect what the object you are currently photographing is. So we planned the "MindSpore Funny Carnival"-to find the most ghostly thing, take this topic and send it to the circle of friends. This is not difficult for many developers, just to allow everyone to get in touch with this product faster and experience its detection speed and accuracy. Overall, we have harvested a lot of funny pictures. And some developers are not satisfied with only using the official apk for scanning, and have retrained the model based on their own data to detect various people on the street, and so on.

University Expansion

The third piece is university expansion. This year, we and the Ministry of Education jointly made a smart base project, signed 72 colleges and universities, will start classes in schools, and MSG colleges and universities activities. Students are the source of users. I hope that students can develop the habit of contributing to the open source community in school, get more exposure to high-quality projects, and continue to improve their technical capabilities.

Enterprise landing

The last piece is the enterprise. For many open source communities, the enterprise sector is actually not easy to do, especially for new open source communities. Our strategy is to hit the head users first. Except for what I showed on the PPT, this year, at HTC (April 23), we and Pengcheng Labs collaborated to produce the Chinese version of the GPT-3 model-Pangu large model. It is also the first and largest Chinese GPT-3 model released in China. There is also the Shenzhen Bay Laboratory. The Shenzhen Bay Laboratory is actually a clue from the community. We did the first Shenzhen MSG on July 20 last year. At that time, Mr. Zhang from the Shenzhen Bay Laboratory came to share with us the content of molecular dynamics. After listening to MindSpore’s sharing, I wondered if I could use MindSpore to do theirs. Training framework, because molecular dynamics simulation has a lot of high-order derivatives to be calculated. They tried to use TensorFlow. At the time, they were using a few versions, which did not support distributed training and calculation of higher-order derivatives. Later, I switched to PyTorch and found it was very useful, but the performance was not as good as TensorFlow. These two are very contradictory. Later, after coming to our MindSpore community, I found that MindSpore has enough advantages in terms of performance, so I used MindSpore to do it, and it is currently progressing smoothly. In addition, it also co-founded the NLP benchmark algorithm with JD.com and applied it in the intelligent customer service scene.

Community governance

Finally, there is the governance of the open source community. Our attitude is completely open source and open. The current technical committee TSC members are from various universities or enterprises. In addition to the expert committee, we also have some SIG groups, which can allow students who are more interested in a certain module to join the SIG group to specifically participate in the development of the work.

You can look at the picture on the left. At present, there are agg map calculation fusion, data data growth and processing, and scurity security, and so on. In fact, there are many SIG groups. This year, we and Teacher Cao of Shanghai Jiaotong University added a new user experience DX-SIG group to collect users' questions and suggestions about using the product, and then make improvements. You can take a look at the development method of the SIG group, and I won't go into details. After submitting an issue, we will have a committer to assign tasks, and after the development is completed, you can submit a PR until it is closed.

Our entire infrastructure includes CI systems, CI-CLA robots and synchronous robots. You can develop on Gitee or contribute to GitHub.

Finally, there is a digital operational kanban. It includes many, including current star, fork, watch, and developers at each level of yours. The changing trend of MindSpore's core indicators, as well as the composition of downloads, user retention, project health, etc., will all be concentrated in the digital operation board. The digital operation Kanban can help us to review the benefits of each activity in time, that is, in addition to our own statistics, we also need a third-party platform to help us make statistics. In this way, both parties can verify that your activities actually bring them. What is the increase in the number of users, and the user’s pull and retention conditions, and then further analysis. All of these can help you to reflect and let you understand what strategies your community should adopt to grow better.

Open source cooperation

For open source cooperation, we have cooperated with Apache TVM, CNCF Foundation, LFAI&Data Foundation, etc. At the same time, there are some specific open source cooperation content in the AIIA Open Source Open Promotion Group, and we will meet with you one by one in the future.

This is the relevant content of MindSpore open source operation and governance that I shared with you today. I hope that more people will join the open source community to operate and promote the open source ecology. Thank you.