Artifact Nginx study manual (recommended collection)

Hello everyone, I'm a migrant worker.

Whether it is operation and maintenance, development, testing, the learning of the Nginx technology stack is always essential, but the depth and breadth of different positions are different. Therefore, the migrant worker took advantage of the weekend break to rearrange the Nginx articles pushed in the past for everyone, and sorted out a systematic Nginx knowledge system for everyone to learn and reference.

The code word is not easy. If you have any help, please click to watch and forward the circle of friends support a wave, thank you! ! ! !

Introduction to Nginx

Nginx is a open source, high-performance, high-reliability Web and reverse proxy server, and supports hot deployment, it can run almost 7 * 24 hours uninterrupted, even if it runs for several months, it does not need to be restarted. The software version can be hot-updated under the condition of uninterrupted service. Performance is the most important consideration for Nginx. It occupies less memory, has strong concurrency capabilities, and can support up to 5w concurrent connections. Most importantly, Nginx is free and can be commercialized, and its configuration and use are relatively simple.

Official website: http://www.nginx.org

Nginx Features

web server

High-performance WEB server software, compared with Apache, it supports more concurrent connections, occupies less server resources, and has high efficiency

Reverse proxy or load balancing server

As a load balancing server, it can be used as a proxy server for services such as HTTP SERVER or DB. Similar to the function of Haproxy proxy software, Nginx's proxy function is relatively simple and not as efficient as Haproxy. It is also an excellent mail proxy service software.

cache server

Nginx can also be used as a cache server, similar to the functions of professional caching software

Nginx pros and cons

Advantages of Nginx

• High concurrency: can support 10,000-20,000 or even more concurrent connections (static small files)
• low memory consumption
• Can do HTTP reverse proxy - load balancing function
• Built-in health check function for cluster node servers, but the function is relatively weak
• Through the cache plug-in, you can realize the functions that the caching software can achieve

Disadvantages of Nginx

Nginx can only support http, https and Email protocols, so the scope of application is smaller, this is its disadvantage

The health check of the backend server only supports detection by port, not by url. Does not support direct hold of Session, but can be solved by ip_hash

Nginx application scenarios

The most important usage scenarios of Nginx:

• Static resource service, providing services through the local file system;
• Reverse proxy service, which extends to include caching, load balancing, etc.;
• API service, OpenResty;

Node.js is no stranger to the front-end. Nginx and Node.js have many similar concepts, such as HTTP server, event-driven, asynchronous non-blocking, etc., and most of the functions of Nginx can also be implemented using Node.js, but Nginx and Node. .js does not conflict, each has its own areas of expertise. Nginx is good at processing low-level server-side resources (static resource processing and forwarding, reverse proxy, load balancing, etc.), Node.js is better at processing upper-layer specific business logic, and the two can be perfectly combined.

Use a diagram to represent:

What exactly can Nginx do? reading this, you will understand !

For more information on Nginx common scenario application configuration, please refer to:

• Please be sure to favorite! Five common application scenarios of
• thoroughly understand the five application scenarios of Nginx! Go out and brag and don't worry anymore

Apache VS Nginx

Both Apache and Nginx are web servers, and both implement the HTTP 1.1 protocol. No matter which one you choose, it is determined according to the application scenario, so these files only start from the application scenario to compare the respective characteristics between the two. Let the right tools do the right things.

Feature comparison

Like Apache, Nginx is an HTTP server software. It adopts a modular structure design in function implementation, and supports common language interfaces, such as PHP, Perl, Python, etc., and also supports forward and reverse proxies, virtual hosts, URL rewriting, compressed transmission, SSL encrypted transmission, etc.

• In terms of functional implementation, all Apache modules support dynamic and static compilation, while Nginx modules are statically compiled.
• For FastCGI support, Apache's support for Fcgi is not good, while Nginx's support for Fcgi is very good;
• In terms of processing connections, Nginx supports epoll, but Apache does not;
• In terms of space usage, the Nginx installation package is only a few hundred K. Compared with Nginx, Apache is definitely a behemoth.

Advantages of Nginx over apache

• Lightweight, same as web service, takes up less memory and resources than apache
• Static processing, Nginx static processing performance is more than 3 times higher than Apache
• Anti-concurrency, nginx processes requests asynchronously and non-blocking, while apache is blocking. Under high concurrency, nginx can maintain low resource consumption and high performance. In - - Apache+PHP (prefork) mode, if PHP processing is slow or front-end pressure is high, the number of Apache processes may soar easily, resulting in denial of service.
• Highly modular design, writing modules is relatively simple
• The community is active, and various high-performance modules are produced quickly

Advantages of apache over nginx

• rewrite, more powerful than nginx's rewrite
• There are so many modules, you can find everything you can think of
• Fewer bugs, nginx has relatively more bugs
• Ultra stable
• Apache's support for PHP is relatively simple, and Nginx needs to be used with other backends

For more detailed comparison instructions, please refer to: Apache VS Nginx, did you choose the right one?

Nginx installation

This article takes the CentOS 7.x system as an example, and uses yum to install Nginx.

yum install nginx -y

After the installation is complete, run the rpm -ql nginx command to view the Nginx installation information.

# Nginx配置文件
/etc/nginx/nginx.conf # nginx 主配置文件
/etc/nginx/nginx.conf.default

# 可执行程序文件
/usr/bin/nginx-upgrade
/usr/sbin/nginx

# nginx库文件
/usr/lib/systemd/system/nginx.service # 用于配置系统守护进程
/usr/lib64/nginx/modules # Nginx模块目录

# 帮助文档
/usr/share/doc/nginx-1.16.1
/usr/share/doc/nginx-1.16.1/CHANGES
/usr/share/doc/nginx-1.16.1/README
/usr/share/doc/nginx-1.16.1/README.dynamic
/usr/share/doc/nginx-1.16.1/UPGRADE-NOTES-1.6-to-1.10

# 静态资源目录
/usr/share/nginx/html/404.html
/usr/share/nginx/html/50x.html
/usr/share/nginx/html/index.html

# 存放Nginx日志文件
/var/log/nginx

There are two folders of primary concern:

• /etc/nginx/conf.d/ is where sub-configuration items are stored, and the main configuration file /etc/nginx/nginx.conf will import all sub-configuration items in this folder by default;
• /usr/share/nginx/html/ static files are placed in this folder, and can also be placed in other places according to your own habits;

Nginx common commands

systemctl system command:

# 开机配置
systemctl enable nginx # 开机自动启动
systemctl disable nginx # 关闭开机自动启动

# 启动Nginx
systemctl start nginx # 启动Nginx成功后，可以直接访问主机IP，此时会展示Nginx默认页面

# 停止Nginx
systemctl stop nginx

# 重启Nginx
systemctl restart nginx

# 重新加载Nginx
systemctl reload nginx

# 查看 Nginx 运行状态
systemctl status nginx

# 查看Nginx进程
ps -ef | grep nginx

# 杀死Nginx进程
kill -9 pid # 根据上面查看到的Nginx进程号，杀死Nginx进程，-9 表示强制结束进程

Nginx application command:

nginx -s reload  # 向主进程发送信号，重新加载配置文件，热重启
nginx -s reopen  # 重启 
Nginxnginx -s stop    # 快速关闭
nginx -s quit    # 等待工作进程处理完成后关闭
nginx -T         # 查看当前 Nginx 最终的配置
nginx -t         # 检查配置是否有问题

Nginx configuration file

The configuration file structure of Nginx is as follows:

# main段配置信息
user  nginx;                        # 运行用户，默认即是nginx，可以不进行设置
worker_processes  auto;             # Nginx 进程数，一般设置为和 CPU 核数一样
error_log  /var/log/nginx/error.log warn;   # Nginx 的错误日志存放目录
pid        /var/run/nginx.pid;      # Nginx 服务启动时的 pid 存放位置

# events段配置信息
events {
    use epoll;     # 使用epoll的I/O模型(如果你不知道Nginx该使用哪种轮询方法，会自动选择一个最适合你操作系统的)
    worker_connections 1024;   # 每个进程允许最大并发数
}

# http段配置信息
# 配置使用最频繁的部分，代理、缓存、日志定义等绝大多数功能和第三方模块的配置都在这里设置
http { 
    # 设置日志模式
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;   # Nginx访问日志存放位置

    sendfile            on;   # 开启高效传输模式
    tcp_nopush          on;   # 减少网络报文段的数量
    tcp_nodelay         on;
    keepalive_timeout   65;   # 保持连接的时间，也叫超时时间，单位秒
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;      # 文件扩展名与类型映射表
    default_type        application/octet-stream;   # 默认文件类型

    include /etc/nginx/conf.d/*.conf;   # 加载子配置项
    
    # server段配置信息
    server {
     listen       80;       # 配置监听的端口
     server_name  localhost;    # 配置的域名
      
     # location段配置信息
     location / {
      root   /usr/share/nginx/html;  # 网站根目录
      index  index.html index.htm;   # 默认首页文件
      deny 172.168.22.11;   # 禁止访问的ip地址，可以为all
      allow 172.168.33.44；# 允许访问的ip地址，可以为all
     }
     
     error_page 500 502 503 504 /50x.html;  # 默认50x对应的访问页面
     error_page 400 404 error.html;   # 同上
    }
}

• main global configuration, effective globally;
• The events configuration affects the network connection between the Nginx server and the user;
• http configuration proxy, cache, log definition and most other functions and configuration of third-party modules;
• The server configures the relevant parameters of the virtual host. There can be multiple server blocks in an http block;
• location is used to configure the matching uri;
• upstream configures the specific address of the back-end server, an indispensable part of the load balancing configuration;

For more information on Nginx configuration files, please refer to the following articles:

• most complete Chinese description of Nginx configuration parameters in history
• Nginx common configuration summary! From entry to work is enough
• Nginx practice: location path matches

If you usually configure Nginx more frequently, I recommend you to use this artifact: powerful! Nginx configuration online one-click generation of "artifact"

Built-in variables commonly used in Nginx

Nginx's process model

Nginx server, during normal operation:

• Multi-process: one Master process, multiple Worker processes
• Master Process: Manage Worker Processes
• External interface: receive external operations (signals)
• Internal forwarding: Manage workers through signals according to external operations
• Monitoring: Monitor the running status of the worker process, automatically restart the worker process after the worker process terminates abnormally
• Worker process: All Worker processes are equal
• Actual processing: network requests, processed by the Worker process;
• Number of Worker processes: Configured in nginx.conf, generally set to the number of cores, to make full use of CPU resources, and at the same time, avoid too many processes, avoid processes competing for CPU resources, and increase the loss of context switching.

Why Nginx is so powerful , you can refer to:

• Why is Nginx efficient? One article to understand the core principle of Nginx
• From the excellent core architecture design of Nginx, reveal why it can support high concurrency?

Nginx optimization configuration

In fact, the core content is mainly tuned by modifying the Nginx configuration file!

• Nginx optimization configuration detailed explanation
• speed up Nginx response!

In our daily work and study, how should we optimize our Nginx server? How should we deal with the following problems?

• How to customize the 404 error page returned to the client
• How to View Server Status Information
• How to optimize Nginx concurrency
• .......

These problems can refer to the solutions of this article: Nginx high performance optimization configuration actual combat summary

How does Nginx achieve concurrency? Why doesn't Nginx use multithreading? What are the common optimization methods for Nginx? What are the possible causes of 502 errors? You must have been asked a interview questions like 161e6e410265a0.

So, How does Nginx achieve high concurrency? What are the common optimization methods?

The security of Nginx server is also very important in daily use. Here I will share with you some practical experience: How to build an efficient and secure Nginx web server

Finally, I will share with you two optimization cases based on HTTPS:

• Nginx-based HTTPS performance optimization practice
• High-performance Nginx HTTPS tuning! Speed up HTTPS by 30%

Nginx log related

After introducing the common scenarios of installation, configuration, and optimization, the log is also very important. Everyone knows that the daily troubleshooting, the log plays a pivotal role.

Nginx logs are mainly divided into two types: access_log (access log) and error_log (error log) . Through the access log, we can obtain the user's IP address, browser information, request processing time and other information. The error log records information about access errors, which can help us locate the cause of the error.

error_log  /var/log/nginx/error.log warn;
#配置错误日志的级别及存储目录

events {
    worker_connections  1024;
}
http {
..................
    log_format  main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    #配置日志的模式
    access_log  /var/log/nginx/access.log main;
    #配置访问日志存储目录
}

For the configuration practice of Nginx log, see this article: Nginx log configuration practice! Super detailed

Due to the powerful functions and outstanding performance of nginx, more and more web applications use nginx as the web server for http and reverse proxy. The access log of nginx is one of the very important data sources for user behavior analysis or security analysis. How to effectively and conveniently collect nginx logs for effective analysis has become a problem that everyone is concerned about.

Through several examples to introduce how filebeat, logstash, rsyslog collection nginx access log and error log , I can: use ELK analysis Nginx log .

Nginx version upgrade

Here, let's talk about how to upgrade Nginx to the new version and how to roll back to the old version, a situation often encountered in the actual production environment of the enterprise.

Version upgrade is actually an upgrade for binary files. The process is as follows:

[root@nginx ~]# /usr/local/nginx-1.12.2/sbin/nginx -v
nginx version: nginx/1.12.2
[root@nginx ~]# cd /usr/local/nginx-1.12.2/sbin/
[root@nginx sbin]# mv nginx nginx-1.12.2
#首先备份原来的旧版本nginx二进制文件
[root@nginx sbin]# cp /usr/local/nginx-1.14.2/sbin/nginx ./
#拷贝新版本的二进制文件到当前目录

Note : In fact, to upgrade the new version, the most important thing is to upgrade smoothly, so that front-end users are not aware (that is, the service is not interrupted, which is actually not difficult, and multiple units in production are upgraded in gray in sequence).

For upgrades, the most difficult thing is not to upgrade, but to roll back, because there is a possibility of rolling back in the actual production environment. For example, the new version is incompatible with existing applications due to some unknown bugs, or the operation is unstable. situation and so on.

So, for us, failback is the point. Here is what I recommend to you: Smooth upgrade and rollback of Nginx version in 1

Nginx reverse proxy and load balancing

Introduction to Reverse Proxy

Reverse Proxy means that the proxy server accepts connection requests on the internet, then forwards the request to the server on the internal network, and returns the result obtained from the server to the client requesting the connection on the internet. At this point, the proxy server acts as a reverse proxy server to the outside world. The reverse proxy serves the server side. The reverse proxy can help the server receive requests from the client, help the server do request forwarding, load balancing, etc.

The reverse proxy is transparent to the server and non-transparent to us, that is, we do not know that we are accessing the proxy server, and the server knows that the reverse proxy is serving him.

Configuration instance

http {
.............
    upstream product_server{
        127.0.0.1:8081;
    }

    upstream admin_server{
        127.0.0.1:8082;
    }

    upstream test_server{
        127.0.0.1:8083;
    }

server {
      
  #默认指向product的server
  location / {
      proxy_pass http://product_server;
      }

  location /product/{
      proxy_pass http://product_server;
     }

  location /admin/ {
      proxy_pass http://admin_server;
     }

  location /test/ {
      proxy_pass http://test_server;
      }
    }
}

If you are not familiar with the location rules in Nginx configuration, I recommend you to read this article: Nginx practice: location path matches . Each location of nginx is a matching directory. The strategy of nginx is: when an access request comes, it will parse the access address, match one by one from top to bottom, and execute the strategy in the corresponding location curly brackets, and according to the strategy request to respond accordingly.

one character "/" in the configuration, the access does not report an error. This problem is one of the reasons for the fault. As the saying goes: 161e6e41026801 An inconspicuous character "/" in the Nginx configuration "The huge role of "missing it by a thousand miles , this article has detailed relevant explanations and examples to verify.

Advantages of reverse proxy:

• hide the real server;
• Load balancing facilitates horizontal expansion of back-end dynamic services;
• Dynamic and static separation, improve system robustness;

Nginx load balancing

nginx can achieve load balancing, what is load balancing? That is to say, the application is deployed on different servers, but is entered through a unified domain name, and nginx distributes the request and distributes the request to different servers for processing, which can effectively reduce the pressure on a single server.

Configuration instance

upstream server_pools { 
  server 192.168.1.11:8880   weight=5;
  server 192.168.1.12:9990   weight=1;
  server 192.168.1.13:8989   weight=6;
  #weigth参数表示权值，权值越高被分配到的几率越大
}
server {  
  listen 80; 
  server_name mingongge.com;
  location / {    
  proxy_pass http://server_pools; 
   }
}

Nginx's strategy for implementing load balancing

• Polling strategy: The strategy adopted by default, assigns all client request polling to the server. This strategy can work normally, but if one of the servers is under too much pressure and there is a delay, it will affect all users assigned to this server.
• Minimum Connections Policy: Prioritize requests to less stressed servers, it can balance the length of each queue and avoid adding more requests to stressed servers.
• Fastest Response Time Policy: Give priority to the server with the shortest response time.
• Client ip binding strategy: requests from the same ip are always allocated to only one server, which effectively solves the problem of session sharing in dynamic web pages.

Want high availability? load balancing architecture is the key ,

For the difference between load balancing and reverse proxy, please refer to: article explains the real difference between load balancing and reverse proxy , about nginx reverse proxy and load balancing strategy Practical case .

Nginx dynamic and static separation

Nginx dynamic and static separation is to allow dynamic web pages in dynamic websites to distinguish constant resources from frequently changing resources according to certain rules. After the dynamic and static resources are split, we can cache them according to the characteristics of static resources. , which is the core idea of website static processing. Recommended: CentOS 7.3: LAMP dynamic and static separation deployment You can learn about the Apache configuration, and then go back and understand Nginx's dynamic and static separation can be more intuitive.

server {  
listen 80;  
server_name mingongge.com;  
location /static {      
  root /wwww/web/web_static_site; 
  }
}

You can also use the following method

location /image {
 alias /web/nginx/static/image/;
}
注意:使用alias末尾一定要添加/,并且它只能位于location中

After the front-end and back-end are separated, the access speed of static resources can be greatly improved. Even if dynamic services are unavailable, the access to static resources will not be affected.

Small scale chopper! Nginx builds a static resource server , this article is to introduce the whole building process, very detailed.

Advanced Nginx Features

In addition to load balancing, Nginx can also do a lot, such as current limiting, caching, black and white lists, etc.

redirect configuration

location / {
 return 404; #直接返回状态码
}
location / {
 return 404 "pages not found"; #返回状态码 + 一段文本
}
location / {
 return 302 /blog ; #返回状态码 + 重定向地址
}
location / {
 return https://www.mingongge.com ; #返回重定向地址
}

Examples are as follows

server { 
listen 80;
server_name www.mingongge.com;
return 301 http://mingongge.com$request_uri;
}
server {
listen 80; 
server_name www.mingongge.com; 
location /cn-url { 
   return 301 http://mingongge.com.cn; 
   }
}
server{
  listen 80;
  server_name mingongge.com; # 要在本地hosts文件进行配置
  root html;
  location /search {
   rewrite ^/(.*) https://www.mingongge.com redirect;
  }
  
  location /images {
   rewrite /images/(.*) /pics/$1;
  }
  
  location /pics {
   rewrite /pics/(.*) /photos/$1;
  }
  
  location /photos {
  
  }
}

traffic copy

requires : Copying the traffic from the production environment to the pre-launch environment or test environment has many benefits, such as:

• It can verify whether the function is normal and the performance of the service;
• Use real and effective traffic requests to verify, without creating data, and without affecting normal online access;
• This is not the same as the grayscale release, and the mirrored traffic will not affect the real traffic;
• Can be used to troubleshoot online problems;
• Refactoring, if the service is refactored, this is also a test method;

In order to realize traffic copying, Nginx provides the ngx_http_mirror_module module, which is another powerful function of Traffic copy

Limiting

The Nginx rate-limiting module uses the leaky bucket algorithm, which can forcefully ensure that the real-time processing speed of requests does not exceed the set threshold.

The official version of Nginx restricts IP connection and concurrency with two modules:

• limit_req_zone is used to limit the number of requests per unit time, that is, rate limit, using the leaky bucket algorithm "leaky bucket".
• limit_req_conn is used to limit the number of connections at the same time, that is, the concurrency limit.

Sunflower Collection! article to get Nginx current limiting configuration

cache

As an important means of performance optimization, Nginx caching can greatly reduce the load on the backend server. Below we will introduce the relevant instructions of Nginx cache configuration and http cache mechanism, as well as Nginx cache practice case analysis.

refer to: 161e6e41026bde Detailed explanation of Nginx caching mechanism!

grayscale release

Implementation process:

• When the user request reaches the front-end proxy service Nginx, the embedded lua module parses the lua script code in the Nginx configuration file;
• The Lua variable obtains the client IP address to query whether the key value exists in the memcached cache. If there is a return value, execute @client_test, otherwise execute @client.
• Location @client_test forwards the request to the server where the new version of the code is deployed, and location @client forwards the request to the server where the normal version of the code is deployed, and the server returns the result. The whole process is complete.

The following is the detailed process of the installation and configuration process: realizes grayscale release based on Nginx+lua+Memcache

Here is an article recommended for you: based Nginx to achieve grayscale publishing and AB testing

Block malicious access

Looking at the access log of nginx, I found that there are many foreign IP addresses visiting my website every day, and the content accessed is basically malicious. So I decided to ban foreign IPs from accessing my website

There are many ways to achieve this function. Next, I will introduce the ngx_http_geoip2 module based on NGINX to prohibit foreign IPs from accessing websites. The detailed solution configuration is as follows: block malicious access through Nginx

Nginx+keepalived achieves high availability

Keepalived software was originally designed for LVS load balancing software to manage and monitor the status of each service node in the LVS cluster system, and later added VRRP (Virtual Router Redundancy Protocol, virtual router redundancy protocol) that can achieve high availability. Features. Therefore, in addition to managing LVS software, Keepalived can also be used as a high-availability solution software for other services (eg: Nginx, Haproxy, MySQL, etc.). keepalived high availability cluster service

For the installation and configuration process, please refer to: Nginx+keepalived to achieve high availability, anti- and dynamic and static separation 161e6e41026d16 configuration, very well written!

The Ultimate Guide to Nginx

Describes 12 operations to improve the security, stability, and performance of Nginx servers.

• Keep Nginx up to date
• Remove unused Nginx modules
• Disable the server_tokens item
• Ban illegal HTTP User Agents
• Disable unwanted HTTP methods
• Set the upper limit of buffer capacity
• Limit the maximum number of connections
• Set up log monitoring
• Block image links
• Disable SSL and only turn on TLS
• Certificate Encryption (HTTPS)
• Redirect HTTP requests to HTTPS

The above 12 specific configuration processes can be referred to: 12 tips to improve the hardness of Nginx server

Nginx monitoring

Monitoring your web server is essential to seeing what is happening on your website. The most concerned is the log changes. When viewing the real-time log file changes, everyone's first reaction should be the 'tail -f /path/to/log' command, but if the access log of each website is viewed in this way, it is quite Crash, today I will share with you a powerful Nginx log monitoring tool.

A small tool to help you get real-time monitoring of Nginx server

After the Nginx cluster is built in the daily production environment, the daily Nginx monitoring needs to be further studied. How does Nginx monitor? I believe that Baidu can find it: nginx-status

After obtaining Nginx monitoring data in real time through Nginx-status, how to integrate with the existing monitoring system? A good solution: Nginx+Telegraf+Influxdb+Grafana

That is, the monitoring status of Nginx is collected regularly through the Telegraf monitoring plug-in, stored in the time series database Influxdb, and then displayed through Grafana.

This is another management artifact I found that can achieve configuration management and performance monitoring. The specific installation and configuration process is in this article: Another Nginx management visualization artifact! Configuration, monitoring one-stop has been introduced.

Nginx knowledge system dynamic update address

Regarding the dynamic update address of the Nginx knowledge system, you can continue to pay attention to the public account of the technology road of migrant workers, and then refer to the Nginx technical column to view the updated articles in real time. Whether it is learning or filling in gaps, it is a very practical technical manual.