Uncomfortable, the memory of nginx worker process continues to float!

background

The first two articles talked about the lua openresty project on the cloud host. After a period of verification in the test environment, everything went smoothly, and the grayscale started online.

However, the good times did not last long. It didn’t take long for the top pod view it. When I checked it with 060c8813cb9de3, I found that the memory was full. At first, I suspected that the allocation of resources limit memory(2024Mi) small. After zooming in (4096Mi), restart the pod, and it was full again soon.

Immediately afterwards, it was suspected to be caused by a large volume and a high load. Expand hpa and restart again. Good guy, it is full again in less than two sticks of incense.

At this point, I turned my attention to the pod where the memory leak occurred.

Where is the pit

1. Every time, the memory goes up and down, and the id number of nginx worker (child process) keeps increasing. Who killed the process?
2. The cloud host does not have such a problem, is it caused by k8s?
3. The increase of pod resources limit memory hpa did not solve the problem.
4. nginx -s reload can release the memory, but it was full after a while.

Solution

Who killed the process?

Command: ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 252672  5844 ?        Ss   Jun11   0:00 nginx: master process /data/openresty/bin/openresty -g daemon off;
nobody     865 10.1  0.3 864328 590744 ?       S    14:56   7:14 nginx: worker process
nobody     866 13.0  0.3 860164 586748 ?       S    15:13   7:02 nginx: worker process
nobody     931 15.6  0.2 759944 486408 ?       R    15:31   5:37 nginx: worker process
nobody     938 13.3  0.1 507784 234384 ?       R    15:49   2:23 nginx: worker process

It is found that the worker process number is close to 1000, so it must be killed continuously and then transferred again. Then who did it? Through the dmesg command:

[36812300.604948] dljgo invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=999
[36812300.648057] Task in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ad18fa_3b8f_4600_a557_a2bc853e80d9.slice/docker-c888fefbafc14b39e42db5ad204b2e5fa7cbfdf20cbd621ecf15fdebcb692a61.scope killed as a result of limit of /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ad18fa_3b8f_4600_a557_a2bc853e80d9.slice/docker-c888fefbafc14b39e42db5ad204b2e5fa7cbfdf20cbd621ecf15fdebcb692a61.scope
[36812300.655582] memory: usage 500000kB, limit 500000kB, failcnt 132
[36812300.657244] memory+swap: usage 500000kB, limit 500000kB, failcnt 0
[36812300.658931] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
……
[36675871.040485] Memory cgroup out of memory: Kill process 16492 (openresty) score 1286 or sacrifice child

It is found that when the cgroup memory is insufficient, the Linux kernel will trigger cgroup OOM to select some processes to kill, so that some memory can be reclaimed, and try to keep the system running.

Although the memory was released after the nginx worker process was killed and the problem was temporarily solved, the fundamental problem has not been solved yet.

Why is there no problem with cloud hosting?

Copy the lua code of the cloud host to the local for comparison, and found that there is no problem with the code itself.

That can only be caused by other issues.

How is it good?

None of the above two points can locate the problem well. It seems that the problem can only be solved by using commands such as top , pmap , gdb

1. Top memory leaked process

Use top see which process occupies high memory

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                               
  942 nobody    20   0  618.9m 351.7m   4.2m S  18.0  0.2   4:05.72 openresty                                                                                                                                                             
  943 nobody    20   0  413.8m 146.7m   4.2m S  11.7  0.1   1:18.93 openresty                                                                                                                                                             
  940 nobody    20   0  792.0m 524.9m   4.2m S   7.0  0.3   6:25.81 openresty                                                                                                                                                             
  938 nobody    20   0  847.4m 580.2m   4.2m S   3.7  0.3   7:15.97 openresty 
    1 root      20   0  246.8m   5.7m   3.9m S   0.0  0.0   0:00.24 openresty                                                                                                                                               

2. pmap view the memory allocation of the process

Through pmap -x pid find out the memory allocation of the process, it is found that 0000000000af7000 has a large memory allocation.

Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000    1572     912       0 r-x-- nginx
0000000000788000       4       4       4 r---- nginx
0000000000789000     148     128     116 rw--- nginx
00000000007ae000     140      28      28 rw---   [ anon ]
0000000000a15000     904     900     900 rw---   [ anon ]
0000000000af7000  531080  530980  530980 rw---   [ anon ]
0000000040048000     128     124     124 rw---   [ anon ]
……

3. /proc/pid/smaps locate the address range of the memory leak

After obtaining the memory address of the large memory, use the cat /proc/pid/smaps command to view the specific starting position of the memory segment:

00af7000-21412000 rw-p 00000000 00:00 0                                  [heap]
Size:             533612 kB
Rss:              533596 kB
Pss:              533596 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:    533596 kB
Referenced:       533596 kB
Anonymous:        533596 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd

4. gcore dumps the process image and memory context

gcore pid

Get the "core.pid" file.

5. gdb load memory information

gdb core.pid

sh-4.2$ gdb core.942
GNU gdb (GDB) Red Hat Enterprise Linux 7.*
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP pid]
Core was generated by `nginx: worker process'.
#0  0x00007ff9435e1463 in ?? ()
"/tmp/core.942" is a core file.
Please specify an executable to debug.
(gdb) 

Enter the above command window

6. dump binary export leaked memory content

At point 3, the memory start address has been obtained, and at this time, our memory address is derived:

dump binary memory worker-pid.bin 0x00af7000 0x21412000

View the exported file size

sh-4.2$ du -sh worker-pid.bin
511M    worker-pid.bin

7. Binary file analysis

Open the binary file with the hex tool and find a large number of json objects in it. By analyzing the json objects, it is found that the so encryption library on the pod is old (not updated on git), and the cloud host relies on the new one.

Replace the so library, restart the pod, the problem is solved! ~.

to sum up

Memory leakage in the production environment is a headache. It is a good idea to analyze it in the above way.