Recommendation system study notes (1)

The origin of the recommendation system, information overload

The emergence and popularization of the Internet has brought a large amount of information to users, which has met their needs for information in the information age. However, with the rapid development of the Internet, the amount of information on the Internet has increased significantly, making users face a large amount of information. When you cannot get the part of the information that is really useful to you, the efficiency of the use of the information is reduced. This is the so-called information overload problem.

Under such circumstances, both information consumers and information producers have encountered great challenges: as information consumers, it is very difficult to find information that interests them from a large amount of information; as information production It is also very difficult to make the information produced by oneself stand out and attract the attention of the majority of users.

In order to solve the problem of information overload, countless scientists and engineers have put forward many genius solutions. In chronological order, the methods are as follows:

• Categories
• search engine

• Recommended system

  ---

  Categories

  definition

  The category catalog is to systematically classify website information and provide a website catalog arranged by category. In each category, the website name, URL link, content summary, and sub-category catalogs belonging to this category are arranged. Browse level by level in the directory to find relevant websites.

  effect

  Categorize web pages for easy search and retrieval

  Conditions of Use

  You need to clearly find the category of the information in advance, if the category is unclear, the corresponding result cannot be found

  Related companies

  Yahoo
  hao123
  
  Journal Navigation
  

  ---

  search engine

  definition

  A search engine refers to a system that collects information from the Internet according to certain strategies and uses specific computer programs, organizes and processes the information, provides users with retrieval services, and displays the retrieved related information to users.

  effect

  Improve the speed at which people obtain and collect information, and provide people with a better network environment.

  Conditions of Use

  Be clear about the information you want to find, know the keywords, and search for what you are looking for. When users cannot find keywords that accurately describe their needs, search engines are powerless

  Related companies

  Google
  
  Baidu
  
  Smart Map Search
  

  ---

  Recommended system

  Common recommendation system

  Taobao
  
  Jingdong
  
  Today's headlines
  

  definition

  A recommendation system is an information filtering system used to predict the user's "rating" or "preference" for items, and to find out in advance the connections that will eventually be generated between the user (User) and the item (Item).

  effect

  When the user does not have a clear demand, it is rooted in the past records, and the information that the user is most interested in is selected and presented from a large amount of information.

  Conditions of Use

  The recommendation system requires existing connections, and predicts future connections from existing connections.

  ---

In-depth understanding of the recommendation system

What is connection

Let’s talk about the word "connection" here. This word has a very broad meaning. Anything that can produce a relationship is connection. For example, the user has made a behavior on the item, or some attributes of the user are the same as the attributes of the item, etc., there is a relationship Just connect.
Why do you say that? This is based on the fact that everything has a general trend of interconnection. For example, people and people tend to have more social connections, so there are various social products; for example, people and goods have more and more consumer connections, so There are various e-commerce products; people and information have more and more reading connections, so there are information flow products.
This is only a purely digital world. As the trend of intelligentization of various physical entities becomes more and more obvious, the Internet of Everything will be further strengthened. The world is a big digital network, but there are only two types of nodes: people and others. People are the ultimate meaning of interconnection. "Others" are collectively referred to as items. Items may be people, information, consumer goods, services, etc. The recommendation system is to continuously discover another type of item nodes that are likely to be connected to people in this huge network, so that they can really establish a connection with the user.

---

Common classifications of recommendation systems

Take watching a movie as an example. Generally speaking, when we want to watch a movie but don't know what to watch, we may use the following methods to decide which movie to watch.

1. Consult a friend. We may open the chat tool, find some good friends who often watch movies, and ask them if they have any movies to recommend. We can even open Weibo, post "I want to watch a movie", and wait for enthusiasts to recommend movies. This method is called social recommendation in the recommendation system, that is, let friends recommend items to themselves.
2. We generally have favorite actors and directors. Some people may open a search engine, enter the name of their favorite actor, and then see if there are any movies in the returned results that they have not watched. For example, I like Zhou Xingchi's movies very much, so I went to Douban to search for Zhou Xingchi, and found that I hadn't watched a movie of his early years, so I would take a look. This way is to look for movies that are similar in content to the movies you have seen before. The recommendation system can automate the above process, find the actors and directors the user likes by analyzing the movies that the user has watched, and then recommend these actors or other movies by the director to the user. This type of recommendation is called content-based filtering in the recommendation system.

3. We may also check rankings, such as the famous IMDB movie rankings, to see what movies other people are watching, what movies other people like, and then find a well-received movie to watch. This approach can be further expanded: if you can find a group of users with similar historical interests and see what movies they are watching recently, then the results may be more in line with your interests than the broad popular list. This approach is called recommendation based on collaborative filtering (collaborative filtering)

   ---

   What is a good recommendation system

   A complete recommendation system generally has 3 participants (as shown in the figure below): users, item providers, and websites that provide the recommendation system.
   

Take book recommendation as an example. First, the recommendation system needs to meet the needs of users and recommend books that interest them.
Second, the recommendation system should allow books from various publishers to be recommended to users who are interested in it, instead of recommending books from a few large publishers.
Finally, a good recommendation system design can allow the recommendation system itself to collect high-quality user feedback, continuously improve the quality of recommendations, and increase
The interaction between users and the website increases the income of the website. Therefore, when evaluating a recommendation algorithm, the interests of the three parties need to be considered at the same time. A good recommendation system is a system that can make the three parties win-win.

However, traditional recommendation systems mostly play a role of "icing on the cake", and generally rarely use it as a core function to carry products. Since the usual goal of recommendation systems is not to help users find relevant content, but to hope that users consume content, the more consumption the better, so the industry has gradually evolved a more deformed understanding, "A good recommendation system should become a time killer. "Users walk in and don't want to come out" is the best.

---

What kind of team is needed to build a recommendation system

There is no upper limit to the complexity of a recommendation system, but there is a minimum standard, so when estimating the size of the recommendation system team below, it is estimated according to the lower limit. The team established in this way is called a "lower limit team".
Here first define the role of the team, since it is to form a "team with a lower limit", of course, according to the principle of saving if you can save.

• Algorithm engineers assume the dual responsibilities of data scientists and machine learning engineers. Their main responsibilities are to clean data, train offline recommendation models, develop algorithm interfaces, and evaluate indicators.
• Back-end development engineer, responsible for the development tasks other than the algorithm, is responsible for invoking the recommended RPC service, developing the necessary filtering logic, API interface development, filling detailed fields, etc.

• The operation and maintenance engineer is responsible for the construction and maintenance of the database, the collection of logs, the high availability of the online system, the collection of user feedback data, and the unified storage of log data.

  ---

This article is based on Xing Wu Dao's recommendation system thirty-six formula and Xiang Liang's recommendation system actual combat.