Ten questions for AI developers: 10 minutes to understand the basic process of AI development

Abstract: answers to AI developers one by one from AI development models, frameworks, tools, and learning methods to improve development efficiency.

This article is shared from Huawei Cloud Community " 10 minutes to understand the basic process of AI development ", author: simple persistence.

1. What is being developed in AI development?

• The result of a hardware developer's work is a server full of integrated circuits, which can run some kind of software when it is turned on.
• The work result of a mobile developer can be a mobile phone APP, including a large amount of binary data, which can be directly run by the Kirin chip.
• The result of an AI developer’s work is a model, which is essentially a function:

f(图片) = "How are you"
f(猫图片) = "Cat"

No matter how complicated this function is, how many variables or parameters there are, AI development is trying to find out the expression of this function.

2. How was the AI model (function) developed?

The AI model is essentially a function. If you want to find an accurate expression of this function, you cannot derive it from logic alone. It is trained. We feed the machine's existing data, and the machine will learn from the data. To find a function that best meets these data (called fitting in technical terms), when new data needs to be predicted, the machine can use this function to predict the result of the new data.

For a person with a certain intelligence ability, generally have the following elements: data + algorithm + model, the development process is to constantly use data and algorithms to make the model more and more close to the real situation, pay attention to approximation rather than derivation, this process Called training.

3. What is the training process of the AI model?

Let's use a model that can distinguish between cat and dog pictures to help understand the training process: "data" means we need to prepare a large number of pictures that have been labeled "cat" or "dog"; "algorithm" refers to us when building the model We plan to use a shallow network or a deep one. If it is a deep layer, how many layers should we use, how many neurons are in each layer, what are the functions, etc., that is, the design of the network architecture, which is equivalent to determining that our prediction function should be What is the general structure.

We use Y=f(W, X, b) to represent this function, X is the existing training data (pictures of cats and dogs), and Y is the label of the existing picture data (the picture is a cat Or a dog), what do W and b mean? At the beginning, we really didn't know that these two parameters need to be found out by ourselves after machine learning, and the process of finding is also the process of training.

"Model" means that we bring data into the algorithm for training, and the machine will continue to learn. When the machine finds the optimal W and b, we say that the model is successfully trained. At this time, our function Y =f(W, X, b) is completely determined

Then we can give the model a new cat or dog picture outside the existing data set, and the model can use the function Y=f(W, X, b) to calculate whether the label of this picture is a cat or Dog, this is reasoning.

A brief summary: whether it is the simplest linear regression model or the more complex deep neural network model with 100 million parameters, the essence is to find a function that can well fit the existing data Y=f(W, X , B), and we hope that this function can also perform well on new unknown data.

4. Why does AI emphasize large amounts of data?

Because only the amount of data is large enough, the model can learn enough and accurately distinguish the characteristics of cats and dogs, and can show sufficient accuracy in the task of distinguishing cats and dogs; of course, when the amount of data is not large, We can also train the model, but the results predicted on the new data set are often much worse.

Assuming that the model for identifying cats needs to be very accurate, it needs photos of cats in various lighting conditions, various backgrounds, and various colors. Not only Persian cats, but Garfield cats, until the model contains all kinds of cats we can think of A face recognition model requires several million photos to be usable. The famous Imagenet dataset contains approximately 14 million images, which greatly facilitates developers.

5. What can a good development tool help developers in the AI development process?

Assuming that there is no ready-made development platform, first you need to learn the commonly used AI development language python, python is the part of the whole process that does not consume energy, and secondly you need to learn traditional machine learning and deep learning related algorithms, which is considered to be the most energy and In terms of time, try to understand and derive the algorithm as much as possible, and then use python code to implement the algorithm and then train the model. The code ability is very important, otherwise your model will not be built.

Then you find a development framework, Huawei MindSpore or Google Tensorflow. You will find that there is no need for such a high mathematical threshold. It does not mean that you need a deep understanding of the algorithm to better train a good model. The model loss function optimization method MindSpore/Tensorflow is built-in Ok, and only a simple call is enough. If MindSpore/Tensorflow is a framework that can greatly reduce the amount of model code, then Keras is a framework that makes the amount of model code as small as amazing, such as the cat and dog classification above. The device, 15 lines of code can be done, dozens of lines of code can write a deep learning network architecture with higher-level optimization methods of convolutional layer, pooling layer and fully connected layer.

A development framework is like the many basic functions provided by the Android framework for mobile APP developers. Without this framework, a simple function of sending pictures to friends, you need to directly deal with the various underlying functional modules of the operating system.

6. Is it enough to have a development framework for AI development?

Of course not. A development framework just helps developers do some parts that developers don’t need to implement, or parts that are too expensive to implement. A complete AI development process includes data preparation, data preprocessing, data labeling, and selection. Algorithms, training, tuning, and deployment are similar to the process of a normal APP development. Just like the production of a car, an automated, high-tech assembly workshop is indispensable, but the stamping and painting workshops also restrict the production efficiency of the car. So what is the biggest headache that restricts the efficiency of AI development? Please see the next question. . What is the biggest difficulty in the AI development process?

7. What is the biggest difficulty in the AI development process?

• To prepare a large amount of data, as mentioned earlier, the more terrifying thing is data labeling. According to statistics, data processing and labeling account for about 70% of the entire development cycle. The so-called labeling is to label the data to tell the machine whether a picture is a cat or a dog. Labeling is a very low-tech job in the eyes of AI developers. It is already a labor-intensive industry. There are a large number of third parties in the industry. The company specializes in data labeling. Take automatic driving as an example. The camera on the front of the car takes real-time pictures of all the pictures in front of the car. The process of automatic driving labeling needs to label the preceding car, obstacles, pedestrians, signal lights, signs, indicator lines, etc. If this process is completely manual It is noted that autonomous driving will always be difficult to commercialize.
• Iterative training is necessary. Training is not done overnight. Various parameters must be modified repeatedly to gradually improve the accuracy of the model. This process currently requires rich experience. If you do not have sufficient computing power, train once, wait three days, and then adjust Participate, wait for another three days, if you want to iterate 20 times, this process is very exhausting. Therefore, even if the cost factor is not considered, computing power is the biggest issue that restricts the popularization of AI.

8. How to solve the problem of AI developers?

Just like developing an office software, it is impossible to use notepad to type code. You need an integrated development environment like visual studio, and a code management tool like Git. Similarly, to develop an AI model, you also need one Integrated development environment, and requires a version management tool.

Provides developers with functions such as code editing environment, grammar checking, and compiler. An AI development platform must also help developers label data, create model training jobs, deploy models and other functions. With a development platform, the development platform must Help developers to finish all hard work and let developers focus on the core code and algorithms.

Of course, it is necessary to provide developers with easy-to-access, high-performance, and low-cost computing power to increase the iteration speed of model training.

9. Is there a way to increase the speed of AI development faster?

Of course there is. There are only two ways to fundamentally improve development efficiency, and to dig to the most essential: reuse and automation, otherwise you can only do it line by line.

To develop an APP, you can start with the most basic code and reuse the development packages, functions, etc. shared by the developers. The Huawei Cloud open source mirror station has a large number of java development packages, which means that you can also reuse a complete APP. , Modify the color of the interface, it becomes your own APP, which is reuse. To ensure the quality of the APP, you can test it manually or use automated testing tools. If you don't want to do this, you can also use a ready-made tool to generate an APP in a visual way. Of course, the style of the APP can only be selected from the existing templates of the tool. Developers have been trying to use various methods to reduce the workload of development.

AI development is the same. You can find ready-made data sets, ready-made labeled data sets, ready-made algorithms and models, and use automatic deployment tools to deploy the models to the end-side cloud. Of course, the automatic learning function can also be used. As long as the original data is provided, the system will help the developer complete the model construction, algorithm selection and automatic training, and the developer can directly see the results. The easiest way is to find a trained model and call it directly in the software.

Based on this, a good AI development platform needs to improve development efficiency from all dimensions. These functions are often many small details, with strong computing power and fast training, but they only complete the most basic functions necessary.

10. What development tools will Huawei provide to developers?

• ModelArts, an AI development platform for AI model development
• HiLens, a visual AI application development platform, is used for AI application development. The models in the application are provided by ModelArts.

The above two products provide services on HUAWEI CLOUD EI in the form of cloud services.

Atlas 200 developer kit (development board) is an offline development kit of ModelArts.

Click to follow and learn about Huawei Cloud's fresh technology for the first time~