Big data surrounds you and me, how can technical people take the road of knowledge sharing

Recommendations of this issue: [Cloud enjoyment person and big coffee face to face] Huawei Cloud Chief Product Official Guowei Exclusive Interview: The current cloud development needs to be broken through is not a technical problem; why is the best time for #华云# now; In response to ever-changing changes, what are the three key starting points for cloud product planning; what is the significance of ecology to the cloud? listen to the technical experts talk about the cloud story.

Abstract: surround you and my life step by step? Zhou Peiyuan, a Huawei Cloud expert who has been in the big data industry for more than five years, has something to say.

This article is shared from the HUAWEI CLOUD community " [Developer who has overcome difficulties] HUAWEI CLOUD·Cloud sharing expert Zhou Peiyuan: Big data surrounds you and me, how do technical people take the road of knowledge sharing ", author: We are all cloud experts.

Since big data was first written into the government work report in 2014, in 7 years, our lives have been surrounded by all kinds of data. Every operation in the network environment has its own data stamp. Coupled with the rapid development of AI, Internet of Things, 5G and other technologies, the value of big data is being maximized and used.

How does this data surround you and my life step by step? Zhou Peiyuan, a Huawei Cloud expert who has been in the big data industry for more than five years, has something to say.

It’s great if the interests coincide with the work

Zhou Peiyuan has been interested in basic computer technology and C++ programming since he was in college. After graduating from college, he chose to work in big data research and development, because "If the interests and work overlap, that would be great."

After several years of big data work, Zhou Peiyuan also has a clear understanding of the development of the domestic big data industry. He enumerated three key factors for the rapid development of this industry.

1. The population is large and the amount of data generated is huge. The seventh national census data shows that the total population of the country is 1.412 billion people. We are socializing, shopping, seeing a doctor, and traveling, and we are constantly generating new data and constructing new data buildings. These data collected through mobile devices, databases, logs, crawlers, etc., will generate huge commercial value after analysis.

2. National policy support. At present, more than 20 regions across the country have issued big data-related policies, and many regions have established special big data management institutions, such as the "Big Data Bureau" in Shanghai and the "Cloud Guizhou" in Guizhou, and universities have also set up big data-related policies. Professional, cultivate a large number of talents for the industry.

3. The development of the Internet industry. What will happen in one minute in China? The mobile payment amounted to 379 million yuan, and 76,000 express delivery items were sent and received... The rapid development of the Internet industry has made big data technology useful. At the same time, big data is also promoting the development of the Internet industry.

In Zhou Peiyuan's view, big data technology can perfectly solve the problems of massive data collection, storage, calculation, and analysis. Its development is to cater to the rigid needs of the Internet era.

And the work experience of the Internet company also enabled Zhou Peiyuan to embark on a special technical road.

"In the beginning, I sorted out the problems and solutions I encountered in my work, and posted them on the blog. I gradually received praise from some fans. At the same time, I also clearly felt the scattered technical points accumulated in my past work. In the process of sorting out It gradually grew into a'tree', and'branches' appeared constantly, which made me feel excited."

After nearly a year of big data technology summarization, Zhou Peiyuan has gained the attention and support of 40,000 fans, and has also been awarded the title of expert by some platforms.

From offline to online, the "magic" of big data

Zhou Peiyuan shared his experience in the construction of several typical big data projects, starting from the business, to dig out the value of data, and to feed back the business.

A second-hand car’s early business volume was small, using a traditional database. With the company’s online construction of data and the increase in business volume, by the end of 2018, the data query volume had surged by 300%, and the database query was delayed or failed, which was no longer sufficient. The needs of headquarters and city operators to view data reports may even affect the normal progress of offline transaction processes.

During the implementation of the project, Zhou Peiyuan designed an incremental calculation plan to compress the calculation time from 1 hour to 5 minutes, and developed a data quality monitoring program to solve the problem of data loss.

After half a year of joint efforts with team members, the project was successfully launched, which effectively guaranteed the normal progress of online business processes, and the available time of the system increased from 95% to 99.99%; it also brought a set of mature real-time calculations to the industry. Solution, and give back to the open source community as a case.

Another case that impressed Zhou Peiyuan was the centralized contract data project of an intermediary platform. The platform opened more than 300 contract centers across the country, but only covered 7% of the contracts. They plan that more than 99% of second-hand housing contracts will be completed through centralized signing. However, there is a lack of data online construction. There are a large number of data indicators missing and access restrictions in the business process. The headquarters and cities cannot accurately evaluate the operation of the contract center, resulting in overall service efficiency. Low.

In response to these pain points, Zhou Peiyuan built a set of centralized contract index system from the three dimensions of quality, scale, and efficiency, and developed a management board for the headquarters and the city to view data indicators. Through these index systems and data reports, the platform has realized effective management of the contract center, realized online and offline integration, intelligent operation, and significantly improved the utilization rate of the venue and the efficiency of personnel.

Knowledge sharing, sparkling a prairie fire

In these years working in big data, Zhou Peiyuan has also clearly felt the changes in the industry. Many traditional enterprises have gradually transformed and upgraded to data-driven enterprises. With the help of big data technology, traditional production, distribution and consumption links have shown unprecedented "informatization", "flattening" and "unbounded". Based on the analysis and research of big data, the prediction of consumer behavior, demand content, structure, mode and its development changes becomes more scientific.

Big data storage and computing technologies are also developing rapidly, from traditional relational databases to distributed databases; from offline batch data extraction to streaming data real-time extraction; from minute-level data query response to second-level query response; from The server is deployed locally to the cloud.

Zhou Peiyuan emphasized that for developers in the front line, they need to continuously improve their technical level to meet the requirements of the Internet industry. "In terms of learning technical knowledge, the Huawei Cloud Developer Community provides us with one-stop services such as resource tools, learning exchanges, application practices, and competition activities. Some tricky technical questions can always be found on the answers or ideas." For example, Huawei There are many free and high-quality learning courses in Cloud Academy. Those who are interested in big data can read the "Python Programming Learning Path", learn through the study + exam, check the gaps, and lay a solid foundation.

Zhou Peiyuan added, "I also often organize and share some big data ecosystem knowledge, technical solutions, programmer stories, etc. on the Huawei Cloud community." (Click to go directly to Zhou Peiyuan's blog homepage)

Finally, sparking a prairie fire, Zhou Peiyuan hopes that through his technology output, he can contribute a little to the vigorous development of the domestic technology community.

Click to follow to learn about Huawei Cloud's fresh technology for the first time~