Get the memory usage of Linux system and Docker container

Recently, I started monitoring the Node.js application we have been developing. After a while, I found that its memory usage increased slowly, increasing by 20% in 3 days. The memory usage is measured in the following Node.js code.

const os = require("os");

const total = os.totalmem();
const free = os.freemem();
const usage = ((free - total) / total) * 100;

So, they basically come from the operating system, in this case Alpine Linux on Docker. Fortunately, I also recorded the memory usage of the application process, but they did not increase. So why does the operating system memory usage increase?

Buffer and cache memory

I used the top command Shift+m (sorted by memory usage) and compared the process on the long-running server with the process on the newly deployed server. The process of both parties is almost the same. The only difference is that buffers and cached Mem are very high in the long run.

After some research or googling, I came to the conclusion that this is not a problem. When the application process requires more memory, most buffers and cached Mem are discarded.

In fact, the free -m command provides a line of used, and free takes buffer and cache considerations.

$ free -m
             total  used  free  shared  buffers cached
Mem:          3950   285  3665     183       12    188
-/+ buffers/cache:    84  3866
Swap:         1896     0  1896

So, what are they? According to the manual /proc/meminfo, this is a pseudo-file and data source free, top and friends:

Buffers %lu
       Relatively temporary storage for raw disk blocks that
       shouldn't get tremendously large (20MB or so).

Cached %lu
       In-memory cache for files read from the disk (the page
       cache).  Doesn't include SwapCached.

I'm still not sure what exactly Buffers contain, but it contains file metadata, etc., and its size is relatively insignificant. Cached contains cached file content, which is called page cache. The operating system reserves the page cache, and RAM has enough free space. This is why even if the process does not leak memory, the memory usage is increasing.

If you are interested, what is the difference between the Buffers and Cached columns in the /proc/meminfo output? There are more detailed information about Buffers and Cached on Quora.

Available memory

So, should we use free + buffers + cached? /proc/meminfo has a better indicator called MemAvailable.

MemAvailable %lu (since Linux 3.14)
       An estimate of how much memory is available for
       starting new applications, without swapping.
$ cat /proc/meminfo
MemTotal:        4045572 kB
MemFree:         3753648 kB
MemAvailable:    3684028 kB
Buffers:           13048 kB
Cached:           193336 kB
...

Its background is well explained in the submission in Linux Kernel, but in essence it excludes non-releasable page caches and includes recyclable tablet memory. The current implementation in Linux v4.12-rc2 still looks almost the same.

Free -m has some implementations of the available column. For example, on Boot2Docker:

$ free -m
       total  used  free  shared  buff/cache  available
Mem:    3950    59  3665     183         226       3597
Swap:   1896     0  1896

It can also be used on AWS CloudWatch metrics via the --mem-avail flag.

Some background on Docker

My other question is "Are these metrics in Docker the same?". Before delving into this issue, let's check how docker works.

According to the Docker overview: the underlying technology, the processes in the Docker container run directly in the host operating system without any virtualization, but due to these Linux kernel features, they are effectively isolated from the host operating system and other containers:

Namespace: Isolate PID, host name, user ID, network access, IPC, etc.

• cgroups: limit resource usage
• UnionFS: Isolated file system
  Due to the namespace, pscommand lists the processes of the Docker container in addition to other processes in the host operating system, but it cannot list the processes of the host operating system or other containers in the docker container.

By default, Docker containers have no resource limits. So, if you run a container on a host and do not limit the resource usage of the container, this is my situation. The "free memory" of the container is the same as the "free memory" of the host operating system.

Memory metrics on Docker containers

If you want to monitor the memory usage of a Docker container from outside the container, this is easy. You can use docker stats.

$ docker stats
CONTAINER     CPU %  MEM USAGE / LIMIT  MEM %  NET I/O     BLOCK I/O  PIDS
fc015f31d9d1  0.00%  220KiB / 3.858GiB  0.01%  1.3kB / 0B  0B / 0B    2

However, if you want to get the memory usage in the container or get more detailed metrics, it becomes complicated. The memory in the Linux container describes the difficulties in detail.

/proc/meminfo and sysinfo, os.totalmem() and os.freemem() are used by Node.js. They are not isolated. If you use top and other common utilities in a Docker container, you will get the host operating system The indicator is free.

To get metrics specific to Docker containers, you can find them in /sys/fs/cgroup/memory/. However, they are not standardized based on the memory in Linux containers.

$ cat /sys/fs/cgroup/memory/memory.usage_in_bytes
303104
$ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712

memory.limit_in_bytes returns a very large number if there is no limit. In this case, you can use /proc/meminfo or use its commands to find the total memory of the host operating system.

in conclusion

This is a longer journey than I initially imagined. My main points are:

• Available Memory> Free Memory means: allowable memory> free memory
• When MemAvailable can be used, use MemAvailable
• The processes in the Docker container run directly in the host operating system
• Know exactly what you are measuring, especially in Docker containers