From the background chat of Netease Cloud Music, how to extract the theme color of the picture

Together with NetEase Cloud🍉

NetEase Cloud Music must be an app that everyone is familiar with. After all, everyone will be

Just kidding, I found a very interesting special effect when I was listening to a song on NetEase Cloud recently:

That is, when switching songs, the background color will be replaced according to the current cover. As a senior Chetuzai, my damn curiosity can't hold back, no, I'm going to find out.

First of all, I conceived a lot of possible ways to implement it:

• Color analysis of images by machine learning
• The front end extracts the main color of the picture and performs gradation
• The cover background image is Gaussian blur

For the first type, he is not within the scope of my knowledge, so I won't expand on it here 😂.

In the second case, it is generally realized canvas

Relatively speaking, from a technical perspective, the third type is the easiest to implement.

After speculation and analysis, I silently opened the familiar Chrome console and opened the source code of NetEase Cloud Music:


Good guy, it really is the third way to achieve it. 🤐

Ben came here, this article should end. But before, a friend asked me how to extract the theme color of the front-end image for I have done a similar requirement before, so let’s expand on it here.

Let's take a picture website as an example to show the most widely used scenarios in actual business:

In a weak network, the image loading speed is slow. At this time, before the image is fully loaded, the main color of the image is extracted, and then the background color is filled. In this way, the user experience can be greatly improved.

How does it work? 🤔

Here we use canvas to achieve, specifically divided into three steps:

• Get image data
• Process image data
• Sort the color list

The test picture we used here is:


Relatively speaking, the main color is more obvious and easy to test~

Get image data🦊

We know that the picture is composed of pixels. The pixel data of the picture can be obtained by the getImageData()

let imgObj = document.getElementById('yourId');

// 创建画布
let canvas = document.createElement('canvas');
canvas.setAttribute('width', imgObj.width);
canvas.setAttribute('height', imgObj.height);
let context = canvas.getContext('2d');
// 将图片画在画布上
context.drawImage(imgObj, 0, 0);
// 获取像素数据
let imgData = context.getImageData(0, 0, imgObj.width, imgObj.height);
let pixelData = imgData.data;

But when you go to print pixelData , you will find the result is:

Good guy, it's all 0,,, 😳

I can't think of the reason for a while: Is it because the canvas api is unskilled?


Found the above answer on stackoverflow

But after I modified it, it still doesn't work.

At this time, I thought that the image loading is asynchronous. It may be that the image data is read from the canvas before the image is loaded. Obviously this is wrong. So I made some adjustments to the original code:

getMainColor("./test.jpeg");
function getMainColor(image) {
  return new Promise((resolve, reject) => {
    try {
      const canvas = document.createElement("canvas");
      const img = new Image(); // 创建img元素
      img.src = image; // 设置图片源地址
      img.onload = () => {
        let color = getImageColor(canvas, img);
        resolve(color);
      };
    } catch (e) {
      reject(e);
    }
  });
}
function getImageColor(canvas, img) {
  const context = canvas.getContext("2d");
  context.drawImage(img, 0, 0);

  // 获取像素数据
  let pixelData = context.getImageData(
    0,
    0,
    canvas.width,
    canvas.height
  ).data;
  console.log("pixelData", pixelData);
  return pixelData;
}

Facts have proved: it's true


Once the image data is acquired, the next step is to process it accordingly.

Process image data🦁

Expand the data obtained in the previous step:

What does the data here mean? In fact, it is rgba , and the distribution represents red (Red), green (Green), blue (Blue) and transparency (Alpha). rgba is represented by the above four values. In other words, every four is a group.

Knowing the law, let us clean up the data: The main thing is to group colors and count the number of times each color appears:

function getImageColor(canvas, img) {
  const context = canvas.getContext("2d");
  context.drawImage(img, 0, 0);

  // 获取像素数据
  let pixelData = context.getImageData(
    0,
    0,
    canvas.width,
    canvas.height
  ).data;
  console.log("pixelData", pixelData);
  return getCountsArr(pixelData);
}
function getCountsArr(pixelData) {
  let colorList = [];
  let rgba = [];
  let rgbaStr = "";
  // 分组循环
  for (let i = 0; i < pixelData.length; i += 4) {
    rgba[0] = pixelData[i];
    rgba[1] = pixelData[i + 1];
    rgba[2] = pixelData[i + 2];
    rgba[3] = pixelData[i + 3];

    if (rgba.indexOf(undefined) !== -1 || pixelData[i + 3] === 0) {
      continue;
    }
    // console.log("rgba", rgba);
    rgbaStr = rgba.join(",");
    if (rgbaStr in colorList) {
      ++colorList[rgbaStr];
    } else {
      colorList[rgbaStr] = 1;
    }
  }
  console.log("colorList", colorList);

  return colorList;
}

The result of colorList


At this point, we have obtained the number of times each type of data appeared.

Sort the color list🐳

The last step is to sort the color value objects obtained above:

for (let prop in colorList) {
  arr.push({
    // 如果只获取rgb,则为`rgb(${prop})`
    color: `rgba(${prop})`,
    count: colorList[prop],
  });
}
// 数组排序
arr.sort((a, b) => {
  return b.count - a.count;
});

console.log("arr", arr);

After sorting, the following results are obtained:


At this point, we have a sorted array of the number of appearances of the image color values from large to small. Let's look at the first rgba(206,205,201,255) :

Paste the test picture again:

The theme colors visible to the naked eye have been extracted! 🎉

Reflection 🚀

Finally, I will return to the player special effects of NetEase Cloud Music mentioned at the beginning of the article. No matter how it is implemented, its product creativity is worth learning.

When we browse some websites at home and abroad or use some apps, we always encounter some effects that make you applaud. These special effects are often inseparable from our front-end.

As the saying goes: The front-end is the development engineer closest to the product. Have you encountered some effects that make you feel amazing or very thoughtful recently? Welcome to leave a message in the comment area🎯

❤️Three Loves

1. If you think this article is not bad, come to share, like, and watch three links, so that more people can see it~

2. Follow the public front-end forest , and regularly push you fresh dry goods and good articles.

3. At special stages, wear a mask and do personal protection.