Lua OpenResty containerization

Original address: Lua OpenResty containerization

background

The company has several projects in the "ancient period", which have always been relatively stable. However, the project always requests QPS per minute to reach the peak value of 800K at some time every day, which leads to some bottlenecks in the performance of the machine. There will be an alert, which is really a headache. What's worse is that this is only one of the projects in ancient times and they are all deployed on physical machines, and all machines add up to close to 100.

For stability (peak clipping) and cost considerations, we finally decided to upload all Lua OpenResty projects to the k8s cluster.

Choose the appropriate openresty base image

By viewing the openresty version information in use online:

/usr/local/openresty/nginx/sbin/nginx -V
nginx version: openresty/1.13.6.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.1.0h  27 Mar 2018 (running with OpenSSL 1.1.0k  28 May 2019)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx ...

lua -v
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio

Learned that openresty/1.13.6.2 and Lua 5.1.4 :

docker pull openresty/openresty:1.13.6.2-2-centos

Q: Can I choose to use a smaller alpine series?

A: Because the project relies on many so libraries, they are all glibc , and for alpine it is musl-lib , which is not compatible.

Q: Why not recompile?

A: On the one hand, it is a risk issue, on the other hand, there are some so libraries that may not be found.

Find the project's dynamic library dependencies

Nginx configuration file

$ tree -L 3 nginx/conf
nginx/conf
├── vhosts/
│    ├── inner.prometheus.nginx.conf
│    └── project.nginx.conf
└── nginx.conf

Self-compiled C dynamic library file, such as binary_protocol.so

Write the dockerfile, then package the project into the container, and execute:

/usr/local/openresty/nginx/sbin/nginx nginx -t

Sure enough, an error was reported:

/usr/local/openresty/nginx/lua/init.lua:1: module 'binary_protocol' not found:
no field package.preload['binary_protocol']
no file '/usr/local/openresty/nginx/lua/binary_protocol.lua'
no file '/usr/local/openresty/nginx/lua_lib/binary_protocol.lua'
no file '/usr/local/openresty/nginx/luarocks/share/lua/5.1/binary_protocol.lua'
no file '/usr/local/openresty/site/lualib/binary_protocol.ljbc'
…… ……
no file '/usr/local/openresty/nginx/luarocks/lib64/lua/5.1/binary_protocol.so'
no file '/usr/local/openresty/site/lualib/binary_protocol.so'
no file '/usr/local/openresty/lualib/binary_protocol.so'
no file '/usr/local/openresty/site/lualib/binary_protocol.so'
no file '/usr/local/openresty/lualib/binary_protocol.so'
no file './binary_protocol.so'
no file '/usr/local/lib/lua/5.1/binary_protocol.so'
no file '/usr/local/openresty/luajit/lib/lua/5.1/binary_protocol.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file '/usr/local/openresty/luajit/lib/lua/5.1/binary_protocol.so'

Q: Observe carefully and find that the so dynamic library is compiled internally and provided for lua to call. How can I find them?

A: It is ldd , pldd or use lsof view the dynamic library file.

Through the ldd and pldd commands, you can view the dependencies related to so

ldd binary_protocol.so
linux-vdso.so.1 =>  (0x00007fff40bd4000)
libtolua++.so => not found        ## 会告诉我们ldd缺少这个依赖
libcrypto.so.6 => not found
liblog4cplus.so.2 => not found        
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f458d9ef000)
libm.so.6 => /lib64/libm.so.6 (0x00007f458d6ed000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f458d4d7000)
libc.so.6 => /lib64/libc.so.6 (0x00007f458d10a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f458df1e000)

Through these methods, a little bit of tracking, knowing that all the dependent libraries can be found.

Luarocks external package file

Find luarocks path included in lua_package_path and lua_package_cpath from nginx.conf online, and then find the manifest file from this path, which describes which luarocks libraries are installed.

luarocks external dependency installation

RUN luarocks --tree=${WORK_DIR}/luarocks install lua-cjson \
    && luarocks --tree=${WORK_DIR}/luarocks install penlight \
    && luarocks --tree=${WORK_DIR}/luarocks install version \
    && luarocks --tree=${WORK_DIR}/luarocks install lua-resty-http \
    && luarocks --tree=${WORK_DIR}/luarocks install luaunit \
    && luarocks --tree=${WORK_DIR}/luarocks install ldoc \
    && luarocks --tree=${WORK_DIR}/luarocks install lua-discount \
    && luarocks --tree=${WORK_DIR}/luarocks install serpent \
    && luarocks --tree=${WORK_DIR}/luarocks install luacov \
    && luarocks --tree=${WORK_DIR}/luarocks install cluacov \
    && luarocks --tree=${WORK_DIR}/luarocks install mmdblua \
    && luarocks --tree=${WORK_DIR}/luarocks install lua-resty-jit-uuid \
    && luarocks --tree=${WORK_DIR}/luarocks install luasocket

RUN luarocks --tree=/usr/local/openresty/nginx/luarocks install nginx-lua-prometheus

Problems encountered and their solutions

Question 1: The container is always OOM Killed

After analysis, it does take up a very large amount of memory:

The number of workers located by the ps command is very large

Solution:

Limited number of workers: worker_processes 4;

Q: Why are there so many workers?

A: On k8s, the worker process started by nginx does not follow the limit we set for the Pod, but is related to the node where the Pod is located.

Question 2: nginx worker process exited on signal 9

Is due to the memory limit set by the Deployment is too small

Solution: requests resource limit of 061af660ee5842

resources:
  limits:
      cpu: "2000m"
      memory: "1Gi"
  requests:
      cpu: "1000m"
      memory: "512Mi"

ps: Starting 4 Workers consumes about 200Mi.

Question 3: attempt to index upvalue'result_dict' (a nil value)

The reason is that the online nginx.conf has related definitions
But there is no code level, just add it:

lua_shared_dict monitor_status 150m;

A small trick to reduce the size of the image

Borrow chicken to

How to access Prometheus monitoring

Access Prometheus in OpenResty, https://github.com/knyar/nginx-lua-prometheus

Installation dependencies

luarocks --tree=/usr/local/openresty/nginx/luarocks install nginx-lua-prometheus

New configuration

Add for nginx/conf/vhosts/project.nginx.conf

lua_shared_dict prometheus_metrics 10M;
log_by_lua_block {
    metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
    metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
}

New configuration file

Added nginx/conf/vhosts/inner.prometheus.nginx.conf

server {
    listen 8099;
    location /metrics {
        content_by_lua_block {
            metric_connections:set(ngx.var.connections_reading, {"reading"})
            metric_connections:set(ngx.var.connections_waiting, {"waiting"})
            metric_connections:set(ngx.var.connections_writing, {"writing"})
            prometheus:collect()
        }
    }
}

Update deployment configuration

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ${name}
  namespace: ${namespace}
  labels:
    test-app: test-server
spec:
  replicas: ${replicas}
  template:
    metadata:
      labels:
        test-app: test-server
      annotations: # <----------------------- 新增
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8099"

Summarize

At this point, a project of Lua has been containerized, and there are still many problems encountered in the middle. Only a few main steps and problems are recorded above.