How are the open source NetEase Cloud Music API projects implemented?

In the previous article, this high-value open source third-party NetEase cloud music player is worth having . An open source third-party NetEase cloud music player is introduced. In this article, let's take a detailed look at the NetEase cloud music used in api The realization principle of the project NeteaseCloudMusicApi .

NeteaseCloudMusicApi is developed using Node.js . There are two main frameworks and libraries, a web application development framework Express , and a request library Axios , these two should be familiar to everyone, so I won't introduce them too much.

Create express app

The entry file of the project is /app.js :

 async function start() {
  require('./server').serveNcmApi({
    checkVersion: true,
  })
}
start()

The serveNcmApi method of the /server.js file is called, let's go to this file, serveNcmApi The method is simplified as follows:

 async function serveNcmApi(options) {
    const port = Number(options.port || process.env.PORT || '3000')
    const host = options.host || process.env.HOST || ''
    const app = await consturctServer(options.moduleDefs)
    const appExt = app
    appExt.server = app.listen(port, host, () => {
        console.log(`server running @ http://${host ? host : 'localhost'}:${port}`)
    })

    return appExt
}

It is mainly to start listening to the specified port, so the main logic of creating an application is in the consturctServer method:

 async function consturctServer(moduleDefs) {
    // 创建一个应用
    const app = express()

    // 设置为true，则客户端的IP地址被理解为X-Forwarded-*报头中最左边的条目
    app.set('trust proxy', true)

    /**
   * 配置CORS & 预检请求
   */
    app.use((req, res, next) => {
        if (req.path !== '/' && !req.path.includes('.')) {
            res.set({
                'Access-Control-Allow-Credentials': true, // 跨域情况下，允许客户端携带验证信息，比如cookie，同时，前端发送请求时也需要设置withCredentials: true
                'Access-Control-Allow-Origin': req.headers.origin || '*', // 允许跨域请求的域名，设置为*代表允许所有域名
                'Access-Control-Allow-Headers': 'X-Requested-With,Content-Type', // 用于给预检请求(options)列出服务端允许的自定义标头，如果前端发送的请求中包含自定义的请求标头，且该标头不包含在Access-Control-Allow-Headers中，那么该请求无法成功发起
                'Access-Control-Allow-Methods': 'PUT,POST,GET,DELETE,OPTIONS', // 设置跨域请求允许的请求方法理想
                'Content-Type': 'application/json; charset=utf-8', // 设置响应数据的类型及编码
            })
        }
        // OPTIONS为预检请求，复杂请求会在发送真正的请求前先发送一个预检请求，获取服务器支持的Access-Control-Allow-xxx相关信息，判断后续是否有必要再发送真正的请求，返回状态码204代表请求成功，但是没有内容
        req.method === 'OPTIONS' ? res.status(204).end() : next()
    })
    // ...
}

First create a Express application, and then set it as a trusted proxy, get it in Express ip usually through req.ip and 86bdf6 req.ips 926d39 req.ips ， trust proxy false ，这种情况下req.ips值是空的，当true时, the value of req.ip 28da88946b61ba8bb30fad56cb44c0ed--- will take the leftmost value from the request header X-Forwarded-For , req.ips will include X-Forwarded-For All ip address.

X-Forwarded-For The format of the header is as follows:

 X-Forwarded-For: client1, proxy1, proxy2

The value is separated by a 逗号+空格 to multiple ip addresses, the leftmost client1 is the original client's ip , Each time a request is successfully received, the source of the request ip address is added to the right.

Taking the above example, the request goes through two proxy servers: proxy1 and proxy2 .请求client1发出， XFF是空的，到proxy1时， proxy1把client1到XFF中，之后请求发往proxy2 ， proxy2的时候， proxy1被添加到XFF , then the request is sent to the final server, and upon arrival proxy2 is added to XFF .

But it is very easy to forge this field, so when the proxy is not trusted, this field is not necessarily reliable, but under normal circumstances XFF the last ip address must be the last proxy server ip address, this will be more reliable.

Then the cross-domain response header is set, and the setting here is the key to allowing websites with different domain names to be able to request successfully.

continue:

 async function consturctServer(moduleDefs) {
    // ...
    /**
   * 解析Cookie
   */
    app.use((req, _, next) => {
        req.cookies = {}
        //;(req.headers.cookie || '').split(/\s*;\s*/).forEach((pair) => { //  Polynomial regular expression //
        // 从请求头中读取cookie，cookie格式为：name=value;name2=value2...，所以先根据;切割为数组
        ;(req.headers.cookie || '').split(/;\s+|(?<!\s)\s+$/g).forEach((pair) => {
            let crack = pair.indexOf('=')
            // 没有值的直接跳过
            if (crack < 1 || crack == pair.length - 1) return
            // 将cookie保存到cookies对象上
            req.cookies[decode(pair.slice(0, crack)).trim()] = decode(
                pair.slice(crack + 1),
            ).trim()
        })
        next()
    })

    /**
   * 请求体解析和文件上传处理
   */
    app.use(express.json())
    app.use(express.urlencoded({ extended: false }))
    app.use(fileUpload())

    /**
   * 将public目录下的文件作为静态文件提供
   */
    app.use(express.static(path.join(__dirname, 'public')))

    /**
   * 缓存请求，两分钟内同样的请求会从缓存里读取数据，不会向网易云音乐服务器发送请求
   */
    app.use(cache('2 minutes', (_, res) => res.statusCode === 200))
    // ...
}

Next, some middleware was registered to parse cookie , process the request body, etc., and also did interface caching to prevent the NetEase cloud music server from being blocked too frequently.

continue:

 async function consturctServer(moduleDefs) {
    // ...
    /**
   * 特殊路由
   */
    const special = {
        'daily_signin.js': '/daily_signin',
        'fm_trash.js': '/fm_trash',
        'personal_fm.js': '/personal_fm',
    }

    /**
   * 加载/module目录下的所有模块，每个模块对应一个接口
   */
    const moduleDefinitions =
          moduleDefs ||
          (await getModulesDefinitions(path.join(__dirname, 'module'), special))
    // ...
}

Next, all modules in the /module directory are loaded:

Each module represents a request to the NetEase Cloud Music interface, such as album_detail.js to get album details:

The module loading method getModulesDefinitions is as follows:

 async function getModulesDefinitions(
  modulesPath,
  specificRoute,
  doRequire = true,
) {
  const files = await fs.promises.readdir(modulesPath)
  const parseRoute = (fileName) =>
    specificRoute && fileName in specificRoute
      ? specificRoute[fileName]
      : `/${fileName.replace(/\.js$/i, '').replace(/_/g, '/')}`
  // 遍历目录下的所有文件
  const modules = files
    .reverse()
    .filter((file) => file.endsWith('.js'))// 过滤出js文件
    .map((file) => {
      const identifier = file.split('.').shift()// 模块标识
      const route = parseRoute(file)// 模块对应的路由
      const modulePath = path.join(modulesPath, file)// 模块路径
      const module = doRequire ? require(modulePath) : modulePath// 加载模块

      return { identifier, route, module }
    })

  return modules
}

Take the album_detail.js module as an example, the returned data is as follows:

 { 
    identifier: 'album_detail', 
    route: '/album/detail', 
    module: () => {/*模块内容*/}
}

The next step is to register the route:

 async function consturctServer(moduleDefs) { 
    // ...
    for (const moduleDef of moduleDefinitions) {
        // 注册路由
        app.use(moduleDef.route, async (req, res) => {
            // cookie也可以从查询参数、请求体上传来
            ;[req.query, req.body].forEach((item) => {
                if (typeof item.cookie === 'string') {
                    // 将cookie字符串转换成json类型
                    item.cookie = cookieToJson(decode(item.cookie))
                }
            })

            // 把cookie、查询参数、请求头、文件都整合到一起，作为参数传给每个模块
            let query = Object.assign(
                {},
                { cookie: req.cookies },
                req.query,
                req.body,
                req.files,
            )

            try {
                // 执行模块方法，即发起对网易云音乐接口的请求
                const moduleResponse = await moduleDef.module(query, (...params) => {
                    // 参数注入客户端IP
                    const obj = [...params]
                    // 处理ip，为了实现IPv4-IPv6互通，IPv4地址前会增加::ffff:
                    let ip = req.ip
                    if (ip.substr(0, 7) == '::ffff:') {
                        ip = ip.substr(7)
                    }
                    obj[3] = {
                        ...obj[3],
                        ip,
                    }
                    return request(...obj)
                })
                // 请求成功后，获取响应中的cookie，并且通过Set-Cookie响应头来将这个cookie设置到前端浏览器上
                const cookies = moduleResponse.cookie
                if (Array.isArray(cookies) && cookies.length > 0) {
                    if (req.protocol === 'https') {
                        // 去掉跨域请求cookie的SameSite限制，这个属性用来限制第三方Cookie，从而减少安全风险
                        res.append(
                            'Set-Cookie',
                            cookies.map((cookie) => {
                                return cookie + '; SameSite=None; Secure'
                            }),
                        )
                    } else {
                        res.append('Set-Cookie', cookies)
                    }
                }
                // 回复请求
                res.status(moduleResponse.status).send(moduleResponse.body)
            } catch (moduleResponse) {
                // 请求失败处理
                // 没有响应体，返回404
                if (!moduleResponse.body) {
                    res.status(404).send({
                        code: 404,
                        data: null,
                        msg: 'Not Found',
                    })
                    return
                }
                // 301代表调用了需要登录的接口，但是并没有登录
                if (moduleResponse.body.code == '301')
                    moduleResponse.body.msg = '需要登录'
                res.append('Set-Cookie', moduleResponse.cookie)
                res.status(moduleResponse.status).send(moduleResponse.body)
            }
        })
    }

    return app
}

The logic is very clear, register each module as a route, and after receiving the corresponding request, pass cookie , query parameters, request body, etc. to the corresponding module, and then request the interface of NetEase Cloud Music , if the request is successful, then process the cookie returned by the NetEase cloud music interface, and finally return all the data to the front end. If the interface fails, then the corresponding processing is also performed.

Which is obtained from the query parameters and request body of the request cookie may not be well understood, because cookie is usually brought from the request body, this should be mainly to support the Node.js Invoke:

After the request is successful, if there is cookie in the returned data, then some processing will be performed. First, if it is a request of https , then SameSite=None; Secure will be set, SameSite is an attribute in Cookie 31e8e458234b3069959ed36546b00cf9--- to restrict third parties Cookie , thereby reducing security risks. Chrome 51 ， CSRF攻击和用户追踪，有三个可选值： strict/lax/none ，默认为lax ， https://123.com的页面里https://456.com域名的接口，默认情况下除了导航到123网址的get Except for requests, other requests will not carry 123 the domain name cookie , if set to strict more strict, will not carry cookie , so this project is set to none in order to facilitate cross-domain calls, and there is no restriction. When set to none , you need to set the Secure attribute.

Finally, write cookie to the front-end browser through the Set-Cookie response header.

send request

Next, let's take a look at the request method used above to send the request. This method is in the /util/request.js file. First, some modules are introduced:

 const encrypt = require('./crypto')
const axios = require('axios')
const PacProxyAgent = require('pac-proxy-agent')
const http = require('http')
const https = require('https')
const tunnel = require('tunnel')
const { URLSearchParams, URL } = require('url')
const config = require('../util/config.json')
// ...

Then there is the specific method of sending the request createRequest , this method is also quite long, let's take a look at it slowly:

 const createRequest = (method, url, data = {}, options) => {
    return new Promise((resolve, reject) => {
        let headers = { 'User-Agent': chooseUserAgent(options.ua) }
        // ...
        })
}

The function will return a Promise , first define a request header object, and add the User-Agent header, this header will save the browser type, version number, rendering engine, and operating system , version, CPU type and other information, the standard format is:

 浏览器标识 (操作系统标识; 加密等级标识; 浏览器语言) 渲染引擎标识 版本信息

Needless to say, forging this header is obviously used to trick the server into thinking the request is coming from the browser and not the server as well.

By default, a few are written dead User-Agent the header is randomly selected:

 const chooseUserAgent = (ua = false) => {
    const userAgentList = {
        mobile: [
            'Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1',
            'Mozilla/5.0 (Linux; Android 9; PCT-AL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.64 HuaweiBrowser/10.0.3.311 Mobile Safari/537.36',
            // ...
        ],
        pc: [
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:80.0) Gecko/20100101 Firefox/80.0',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0',
            // ...
        ],
    }
    let realUserAgentList =
        userAgentList[ua] || userAgentList.mobile.concat(userAgentList.pc)
    return ['mobile', 'pc', false].indexOf(ua) > -1
        ? realUserAgentList[Math.floor(Math.random() * realUserAgentList.length)]
    : ua
}

Keep reading:

 const createRequest = (method, url, data = {}, options) => {
    return new Promise((resolve, reject) => {
        // ...
        // 如果是post请求，修改编码格式
        if (method.toUpperCase() === 'POST')
            headers['Content-Type'] = 'application/x-www-form-urlencoded'
        // 伪造Referer头
        if (url.includes('music.163.com'))
            headers['Referer'] = 'https://music.163.com'
        // 设置ip头部
        let ip = options.realIP || options.ip || ''
        if (ip) {
            headers['X-Real-IP'] = ip
            headers['X-Forwarded-For'] = ip
        }
        // ...
    })
}

继续设置了几个头部字段， Axios json ， POST application/x-www-form-urlencoded格式.

Referer头代表发送请求时所在页面的url ， https://123.com页面内https://456.com ， Referer The header will be set to https://123.com , this header is generally used for anti-leech. So forging this header is also to trick the server that the request is coming from their own page.

Next, two ip headers are set, realIP need to be passed manually by the front end:

continue:

 const createRequest = (method, url, data = {}, options) => {
    return new Promise((resolve, reject) => {
        // ...
        // 设置cookie
        if (typeof options.cookie === 'object') {
            if (!options.cookie.MUSIC_U) {
                // 游客
                if (!options.cookie.MUSIC_A) {
                    options.cookie.MUSIC_A = config.anonymous_token
                }
            }
            headers['Cookie'] = Object.keys(options.cookie)
                .map(
                (key) =>
                encodeURIComponent(key) +
                '=' +
                encodeURIComponent(options.cookie[key]),
            )
                .join('; ')
        } else if (options.cookie) {
            headers['Cookie'] = options.cookie
        }
        // ...
    })
}

Next, set cookie , which is divided into two types, one is the object type, in this case cookie generally comes from query parameters or request body, and the other is a string, this is Under normal circumstances, the request header is brought over. MUSIC_U登录后的cookie ， MUSIC_A token ，未登录情况下调用某些接口可能报错， So a visitor token will be set up:

continue:

 const createRequest = (method, url, data = {}, options) => {
    return new Promise((resolve, reject) => {
        // ...
        if (options.crypto === 'weapi') {
            let csrfToken = (headers['Cookie'] || '').match(/_csrf=([^(;|$)]+)/)
            data.csrf_token = csrfToken ? csrfToken[1] : ''
            data = encrypt.weapi(data)
            url = url.replace(/\w*api/, 'weapi')
        } else if (options.crypto === 'linuxapi') {
            data = encrypt.linuxapi({
                method: method,
                url: url.replace(/\w*api/, 'api'),
                params: data,
            })
            headers['User-Agent'] =
                'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36'
            url = 'https://music.163.com/api/linux/forward'
        } else if (options.crypto === 'eapi') {
            const cookie = options.cookie || {}
            const csrfToken = cookie['__csrf'] || ''
            const header = {
                osver: cookie.osver, //系统版本
                deviceId: cookie.deviceId, //encrypt.base64.encode(imei + '\t02:00:00:00:00:00\t5106025eb79a5247\t70ffbaac7')
                appver: cookie.appver || '8.7.01', // app版本
                versioncode: cookie.versioncode || '140', //版本号
                mobilename: cookie.mobilename, //设备model
                buildver: cookie.buildver || Date.now().toString().substr(0, 10),
                resolution: cookie.resolution || '1920x1080', //设备分辨率
                __csrf: csrfToken,
                os: cookie.os || 'android',
                channel: cookie.channel,
                requestId: `${Date.now()}_${Math.floor(Math.random() * 1000)
                .toString()
                .padStart(4, '0')}`,
            }
            if (cookie.MUSIC_U) header['MUSIC_U'] = cookie.MUSIC_U
            if (cookie.MUSIC_A) header['MUSIC_A'] = cookie.MUSIC_A
            headers['Cookie'] = Object.keys(header)
                .map(
                (key) =>
                encodeURIComponent(key) + '=' + encodeURIComponent(header[key]),
            )
                .join('; ')
            data.header = header
            data = encrypt.eapi(options.url, data)
            url = url.replace(/\w*api/, 'eapi')
        }
        // ...
    })
}

This piece of code will be difficult to understand, and the author did not understand it. Anyway, this project uses four types of NetEase Cloud Music interfaces: weapi , linuxapi , eapi , api , for example:

 https://music.163.com/weapi/vipmall/albumproduct/detail
https://music.163.com/eapi/activate/initProfile
https://music.163.com/api/album/detail/dynamic

Each type of interface request parameters and encryption methods are different, so they need to be processed separately:

For example weapi :

 let csrfToken = (headers['Cookie'] || '').match(/_csrf=([^(;|$)]+)/)
data.csrf_token = csrfToken ? csrfToken[1] : ''
data = encrypt.weapi(data)
url = url.replace(/\w*api/, 'weapi')

Add the value of cookie in _csrf to the request data, and then encrypt the data:

 const weapi = (object) => {
  const text = JSON.stringify(object)
  const secretKey = crypto
    .randomBytes(16)
    .map((n) => base62.charAt(n % 62).charCodeAt())
  return {
    params: aesEncrypt(
      Buffer.from(
        aesEncrypt(Buffer.from(text), 'cbc', presetKey, iv).toString('base64'),
      ),
      'cbc',
      secretKey,
      iv,
    ).toString('base64'),
    encSecKey: rsaEncrypt(secretKey.reverse(), publicKey).toString('hex'),
  }
}

Check out other encryption algorithms: crypto.js .

As for how to know these, either it is an insider of NetEase Cloud Music (basically impossible), or it is reversed, such as the interface of the web version, open the console, send a request, find the location in the source code, and break the point , check the request data structure, read the compressed or obfuscated source code and try slowly, in short, pay tribute to these big guys.

continue:

 const createRequest = (method, url, data = {}, options) => {
    return new Promise((resolve, reject) => {
        // ...
        // 响应的数据结构
        const answer = { status: 500, body: {}, cookie: [] }
        // 请求配置
        let settings = {
            method: method,
            url: url,
            headers: headers,
            data: new URLSearchParams(data).toString(),
            httpAgent: new http.Agent({ keepAlive: true }),
            httpsAgent: new https.Agent({ keepAlive: true }),
        }
        if (options.crypto === 'eapi') settings.encoding = null
        // 配置代理
        if (options.proxy) {
            if (options.proxy.indexOf('pac') > -1) {
                settings.httpAgent = new PacProxyAgent(options.proxy)
                settings.httpsAgent = new PacProxyAgent(options.proxy)
            } else {
                const purl = new URL(options.proxy)
                if (purl.hostname) {
                    const agent = tunnel.httpsOverHttp({
                        proxy: {
                            host: purl.hostname,
                            port: purl.port || 80,
                        },
                    })
                    settings.httpsAgent = agent
                    settings.httpAgent = agent
                    settings.proxy = false
                } else {
                    console.error('代理配置无效,不使用代理')
                }
            }
        } else {
            settings.proxy = false
        }
        if (options.crypto === 'eapi') {
            settings = {
                ...settings,
                responseType: 'arraybuffer',
            }
        }
        // ...
    })
}

Here mainly defines the data structure of the response, defines the configuration data of the request, and does some special processing for eapi , the most important thing is the related configuration of the proxy.

Agent is the Node.js HTTP class in the module responsible for managing http persistent connections with client-side connections. It maintains a queue of pending requests for a given host and port, reusing a single socket connection for each request until the queue is empty, at which point the socket is either destroyed or put into a pool where it will be It is used again to request the same host and port. In short, it saves the time to recreate the socket every time a request is initiated http and improves efficiency.

pac refers to the proxy auto-configuration, which actually contains a text file of the javascript function. This function will decide whether to connect directly or through a proxy, which is more convenient than writing a proxy directly. One point, of course, what needs to be configured options.proxy is the remote address of this file, the format is: 'pac+【pac文件地址】+' . pac-proxy-agent模块会http.Agent 78eb12e4d8d740c6593bcb63c0ab42db---实现，它会根据指定的PAC代理文件HTTP 、 HTTPS --OR HTTPS SOCKS Proxy, or direct connection.

As for why to use the tunnel module, the author searched and still did not understand, it may be the solution http protocol interface request Netease cloud music https protocol interface Failed question? Friends who know can explain it in the comment area~

at last:

 const createRequest = (method, url, data = {}, options) => {
    return new Promise((resolve, reject) => {
        // ...
        axios(settings)
            .then((res) => {
                const body = res.data
                // 将响应的set-cookie头中的cookie取出，直接保存到响应对象上
                answer.cookie = (res.headers['set-cookie'] || []).map((x) =>
                    x.replace(/\s*Domain=[^(;|$)]+;*/, ''),// 去掉域名限制
                )
                try {
                    // eapi返回的数据也是加密的，需要解密
                    if (options.crypto === 'eapi') {
                        answer.body = JSON.parse(encrypt.decrypt(body).toString())
                    } else {
                        answer.body = body
                    }
                    answer.status = answer.body.code || res.status
                    // 统一这些状态码为200，都代表成功
                    if (
                        [201, 302, 400, 502, 800, 801, 802, 803].indexOf(answer.body.code) > -1
                    ) {
                        // 特殊状态码
                        answer.status = 200
                    }
                } catch (e) {
                    try {
                        answer.body = JSON.parse(body.toString())
                    } catch (err) {
                        answer.body = body
                    }
                    answer.status = res.status
                }
                answer.status =
                    100 < answer.status && answer.status < 600 ? answer.status : 400
                // 状态码200代表成功，其他都代表失败
                if (answer.status === 200) resolve(answer)
                else reject(answer)
            })
            .catch((err) => {
                answer.status = 502
                answer.body = { code: 502, msg: err }
                reject(answer)
            })
    })
}

The last step is to use Axios to send the request, process the response cookie , save it to the response object for subsequent use, and process some status codes, you can try-catch is used a lot. As for why, it is estimated that you need to try more to know what went wrong. If you are interested, you can try it yourself.

Summarize

This article understands the implementation principle of the NeteaseCloudMusicApi project from the perspective of source code, and we can see that the whole process is relatively simple. It is nothing more than a request proxy. The difficulty lies in finding these interfaces and reversely analyzing the parameters, encryption methods, and decryption methods of each interface. Finally, I would like to remind you that this project is only for learning and use, please do not engage in commercial activities or violate copyright~