node project from 0 to 1 actual combat

foreword

This article takes the koa framework as an example to build a back-end service from 0 to 1, covering the basic content of the back-end service and realizing the functions of api.

Suitable for the crowd:

1. People who have not fully built node services
2. Learned to write koa, those who want to practice
3. People who are using koa to build node services

text

In order to implement the api interface request, we consider several issues:

1. The processing flow and error handling of the entire service program
2. interface routing
3. API call permission
4. interface cache
5. access database

In addition to the service program itself, we must also consider engineering-related (this article will not expand on it):

• log processing
• Monitoring alarm
• quick recovery

Get to know common middleware

request parameter parsing plugin

There are 3 kinds of parameters:

• url search
• url parameter
• POST body

koa-bodyparser : Mount the data on the request body to ctx.request.body, support json / text / xml / form (does not support multipart)

file cache related

• koa-static : Static file system, supports maxage , gzip and other attributes. This middleware works better with the following middleware:

  • Use koa-conditional-get for freshness detection and koa-etag for negotiation cache
  • Use with koa-mount for path control, for example, when accessing /public, the file content is returned
• koa-mount : Multiple child applications are combined into one parent application. (It can also be used to control the mount of middleware through path)

• koa-conditional-get : enable negotiation cache (304 verdict)

  • Cache mechanism: https://juejin.cn/post/6844904133024022536
  • Browser cache: https://segmentfault.com/a/1190000008377508
• koa-etag : support etag/if-none-match negotiation cache

interface cache

Interface caching needs to be done with Redis to implement interface caching.

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker.

Here is an npm used by Node: ioredis
It also needs to be paired with koa-conditional-get and koa-etag to implement the entire cache process.
Use the cache demo , the main knowledge points:

1. Cache settings:

     if (ttl) {
    ctx.response.set('Cache-Control', `max-age=${ttl}`);
     } else {
    ctx.response.set('Cache-Control', 'no-store');
     }

2. Generate redis key: method + url + request body

   const key = `spacex-cache:${hash(`${method}${url}${JSON.stringify(ctx.request.body)}`)}`;

HTTP security

koa-helmet : helmet makes applications more secure by setting Http headers.
Reference: https://juejin.cn/post/6844903699584647175

CORS

koa-cors : CORS (Cross-Origin Resource Access) settings
There are several key header settings for cross-origin resource access:

• Access-Control-Allow-Credentials
• Access-Control-Allow-Origin
• Access-Control-Allow-Headers
• Access-Control-Allow-Methods
• Access-Control-Max-Age

debug

• koa-pino-logger： logger middleware

login design

Use token or session-cookie?

• token: heavy calculation, light storage
• session: heavy storage, light calculation

details, click here >> We use token verification as an example here.

token implementation

Conditions that the token needs to meet

1. Unique ID, representing a unique user account
2. Validity period, after the expiration, you need to log in again to protect the user account

shabby implementation

1. Unique ID through UUID
2. Equivalent token validity period through Redis cache validity period

elegant implementation

Using jsonwebtoken (JWT): https://github.com/auth0/node-jsonwebtoken
Features:

1. Encryption/Decryption Mechanism
2. Generate unique ID
3. Can support validity period setting

For example 🌰 :
The server generates the token:

const token = jwt.sign(
{ // 加密参数
  username:'myName',
  password:'myPassword'
}, 
'MY_SECRET',  // 密码
{ 
  expiresIn: 60 * 60 // 设置有效期
 }
);

token: something like:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsZGFwIjoiemh1YmVubGVpIiwicGFzc3dvcmQiOiJ6MTM2NjU0Mjc4NzEiLCJpYXQiOjE2NDM0NTEzNDQsImV4cCI6MTY0NDA1NjE0NH0.1aewCZmMIkQWoJiZWmdobcPwGY7BuPzWMygf3aw7Z6g

The server parses the token:

const decoded = jwt.verify(token,'MY_SECRET');
console.log(decoded);
// 输出结果
// { 
//  username:'myName',
//  password:'myPassword'
// }

We found that the token can also bring its own parameters, which can save the step of storing user information in the database, but only consume performance during calculation.

Implementation in application:

1. First, get a new token and store it locally
2. Each request carries the token in the x-access-token

// config.js
export const LOCAL_KEY = `${你的域名}-token`; // 这样避免重复

// request.js
const axiosInstance: AxiosInstance = axios.create(requestConfig);
axiosInstance.interceptors.request.use(config => {
  config.headers['x-access-token'] = localStorage.getItem(LOCAL_KEY) || ''; // 带上 token(不要把这个设置放到axios.create 里面： 不会实时更新)
  // 在发送请求之前做些什么
  return config;
}, error => {
  // 对请求错误做些什么
  return Promise.reject(error);
});
...

Summarize

This section introduces the generation, parsing and actual use of front-end http requests on the server side token .

Design of user information

The design of user information should distinguish user rights design

User Info

User information is more general information, which only contains pure user information, such as username, ldap, password, and avatar.
User table design:

username	ldap	password	avatar
Zhang San	zhangsan01	z123456	https://avatar.com/z123456
Li Si	lisil000	l123456	https://avatar.com/l000

User rights

User permissions are the enhancement of user information, and each user is associated with a series of permissions. User permissions can be considered as the realization of the business side, with richer information and heavier business.
Permission table design:

ldap	product	permission
zhangsan01	product	13
zhangsan01	product	21
lisi	product	17

User Identity Validation/Invalidation Mechanism

Effective:

• Regenerate the token when logging in
• Regenerate the token when the password is changed

fail:

• when logging out
• token expired

Follow-up processing:

1. Check the redirect address to jump after re-login
2. When you change your password and it expires, you should be directed to log in again.

3. After logging out

   • For the local way of token, just delete the local token directly, and then proceed to step 2
   • For session mode, you need to invalidate sessionId

interface design

interface design

I won't go into it here, you can refer to Restful Api

Interface verification

• Which interfaces need to verify user identity and how to verify
• Which do not require verification and how to skip verification
• How to pass user information in a transaction after obtaining user information

Auth explanation

Click here for auth in the demo >> This example is used in routing, we will use another method, write it in the outermost layer, to achieve on-demand verification. This avoids introducing this middleware everywhere it is used.
koa-unless : Conditionally skip a middleware when a condition is met.

auth middle file:

var verifyToken = async(ctx, next) => {
  const req = ctx.request;
  const token = req.body.token || req.query.token || req.headers["x-access-token"];
};
  if (!token) {
    ctx.body = {
      code: 0,
      err_code: 401,
      err_msg:'401'
    };
  }else{
    try {
      const decoded = jwt.verify(token, TOKEN_KEY);
      req.user = decoded; // 挂载数据
      await next();
    } catch (err) {
      ctx.body = {
        code: 0,
        err_code: 401,
        err_msg:'401'
      }
    }
  }
};

verifyToken.unless = require('koa-unless');
module.exports = verifyToken；

app.js

const verifyToken = require('middleware/auth');
...
// 身份验证
app.use(verifyToken.unless({
  path: [ // 设置不使用 auth 中间件的 path
    /\/login/, // 登录使用的接口
  ],
}));
// 进入路由处理
app.use(routes());
...

User identity pass:

module.exports = async (ctx, next) => {
  const key = ctx.request.headers['spacex-key'];
  if (key) {
    const user = await db.collection('users').findOne({ key });
    if (user?.key === key) {
      ctx.state.roles = user.roles; // 挂载到 ctx.state上，传递到后面的中间件
      await next();
      return;
    }
  }
  ctx.status = 401;
  ctx.body = 'https://youtu.be/RfiQYRn7fBg';
};

routing design

need to be considered

• Restful Design Approach
• Processing without permission
• Interface structure & error message design

use routing

use koa-route
Refer to demo :

• Sub-module management api, the overall export of the entry file
• used auth middleware
• Use ORM syntax and modlel [covered in this article]
• Use Redis to do interface cache

Database connection (MYSQL version)

First Edition: Handwritten SQL Statements Using koa-mysql

The problem is: you need to abstract the sql syntax yourself. Because sql statements can be abstracted according to their functions (such as abstract conditional query), if they are all written by hand, they will be written more.

// from: https://chenshenhai.github.io/koa2-note/note/mysql/info.html

const mysql = require('mysql')
// 创建数据池
const pool  = mysql.createPool({
  host     : '127.0.0.1',   // 数据库地址
  user     : 'root',    // 数据库用户
  password : '123456'   // 数据库密码
  database : 'my_database'  // 选中数据库
})

// 在数据池中进行会话操作
pool.getConnection(function(err, connection) {

  connection.query('SELECT * FROM my_table',  (error, results, fields) => {

    // 结束会话
    connection.release();

    // 如果有错误就抛出
    if (error) throw error;
  })
})

Second Edition: Using sequlize (orm)

What is ORM ? ORM is the technology to complete the operation of relational database through the syntax of instance object. Representatives are: sequelize / openrecord / typeorm

shortcoming:

• Performance issues -> You don't need to consider this problem if you don't write particularly complex or special SQL
• Writing SQL in an object-oriented way always feels weird. -> Habit problem

Database information storage [The author has not solved it]

• Username and password storage: how to store it securely without revealing the password when using it?
• Database permissions issue: connection administrator or normal user?

overall order

handle cross domain

The benefits of restricting cross-domain:

• Prevent being called on other websites and control the amount of requests
• Prevent using interface tools to call
• With user authentication, the amount of requests can be further controlled (no account can not be accessed)

// 检查 referer, 防止 postman 这种调用
app.use(async (ctx, next) => {
  const { referer = ''} = ctx.header;
  if(ENV === 'production' && !referer.includes(HOST)){
    ctx.response.body = 'Not Allow Origin Request';
  }else{
    await next();
  }
})
// cors
app.use(cors({
  origin:(ctx) => { // 设置可访问这个服务的来源域
    return ENV === 'development' ? 'http://127.0.0.1:8080' : 'https://www.xxx.com';
  },  
  credentials: true,
  allowMethods: ['GET', 'POST', 'PUT','PATCH', 'DELETE'],
  allowHeaders: [
    'Content-Type', 
    'Accept', 
  ],
}));

app.js

// 1. 创建 app
app = new Koa();

//2. 加载辅助中间件
app.use(conditional());
app.use(etag());
app.use(bodyParser());
app.use(helmet());
... // 其他中间件

// 3. 域名检查
app.use(referer()) // referer 验证
app.use(cors());

// 4. 用户身份检查
app.use(verifyToken.unless({
  path: [
    /\/login/
  ],
}));

// 5. 进入路由
app.use(routes());

// 0. 监听 port
app.listen(PORT, () => {
    console.log(`port ${PORT} is listening ~`);
});

error handling

• uncaughtException
• unhandledRejection

// gracefulShutdown: 关机程序。可以理解为遇到错误时候的统一处理
// Server start
app.on('ready', () => {
  SERVER.listen(PORT, '0.0.0.0', () => {
    logger.info(`Running on port: ${PORT}`);

    // Handle kill commands
    process.on('SIGTERM', gracefulShutdown);

    // Handle interrupts
    process.on('SIGINT', gracefulShutdown);

    // Prevent dirty exit on uncaught exceptions:
    process.on('uncaughtException', gracefulShutdown);

    // Prevent dirty exit on unhandled promise rejection
    process.on('unhandledRejection', gracefulShutdown);
  });
});

Reference: https://github.com/r-spacex/SpaceX-API SpaceX code
error middleware: https://sourcegraph.com/github.com/r-spacex/SpaceX-API/-/blob/middleware/errors.js
logger middle (for debugging) : https://sourcegraph.com/github.com/r-spacex/SpaceX-API/-/blob/middleware/logger.js

Deploy to remote server

GitHub Action More practice reference here: Github Action Workflow Practice

Fast recovery of server services: pm2 deploy

Advantage:

• Monitor file changes and automatically restart the program
• Support performance monitoring [also important]
• load balancing
• Program crashes and restarts automatically [Important]
• Automatically restart when the server restarts [Important]
• Automated deployment projects

Automatic deployment to remote servers

Server Conditions:

• The service code is cloned to /data/server/crm-server , so that only git pull is needed each time.
• install node
• Install PM2 globally

name: 服务部署
on: 
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: executing remote ssh commands using password
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.HOST }}
          username: ${{ secrets.USERNAME }}
          password: ${{ secrets.PASSWORD }}
          port: ${{ secrets.PORT }}
          script: |
            cd /data/server/crm-server
            git checkout main
            git pull
            # 如果你的远程服务器是用nvm装的node,需要下面的 export
            export PATH=$PATH:/home/ubuntu/.nvm/versions/node/v16.5.0/bin
            pm2 link 你的pm2链接 # 【可选】添加 pm2监控
            pm2 restart start.sh # 启动 pm2

start.sh finally executes:

$ cross-env PORT=8080 ENV=production node app.js

nginx configuration [optional]

For more detailed nginx configuration, please refer to nginx knowledge that the front end should master

I have deployed both the front-end file (under the /data/www folder) and the back-end API to the same server, taking http://www.ddup.info as an example:

server {
  ...

    location ^~ /crm/api {
      proxy_pass http://www.ddup.info:8080;
    }

    location ^~ /crm {
      root /data/www;
      index index.html index.htm;
      try_files $uri $uri/ /crm/index.html;
    }

    location / {
      root /data/www;
      index index.html index.htm;
      try_files $uri /app/index.html;
    }
}

Thinking: A Reasonable Backend Engineering Structure

• Reference express Directory structure: https://github.com/expressjs/express/tree/master/examples/mvc
• Refer to the egg directory structure: https://eggjs.org/zh-cn/basics/structure.html

Directory Structure:

• static files: static
• view layer: ejs template
• Data model: models✅
• service: service
• Routing: ✅ routes
• Middleware: middleware✅
• Scheduled tasks: jobs ✅

postscript

In the first attempt, it is inevitable that there are ill-considerations. I also ask readers to point out and learn and progress together~