Technology Sharing | ClickHouse GDB debugging notes

Author: xuty

Source of this article: original submission

* Produced by the Aikesheng open source community, original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.

---

1. Background

Record the process of using GDB to debug ClickHouse source code for the first time. This is just a simple debugging process to understand the internal mechanism of ClickHouse, which will help solve the doubts. The code is white, and if there are errors, please forgive me.

2. Debugging issues

Debugging ClickHouse is mainly to solve a practical problem encountered by individuals. The problem is described below:

1. When importing full data through the mysql table function that comes with clickhouse (a test table memory_test , 50w row data 56G was built here), because the maximum memory limit ( CK server 36G memory) was exceeded, the following error was reported.

localhost :) insert into `test`.`memory_test`  select * mysql('192.168.1.1:3306','test','memory_test','root','xxxx');                                                  

Received exception from server (version 20.8.12):
Code: 241. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Memory limit (total) exceeded: would use 35.96 GiB (attempt to allocate chunk of 17181966336 bytes), maximum: 28.14 GiB: While executing SourceFromInputStream. 

2. The general process is that ClickHouse will read mysql data into memory and then write them in batches. This involves a buffer problem. ClickHouse will read rows of data or bytes of data and then write them in batches. According to the observation of show processlist It seems that more than 100 w rows of data will be written once, then the memory is released, and the reading and writing are repeated. So our problem actually reads more than 1 million rows original MySQL table, which exceeds the size of our ClickHouse memory configuration.

3. Regarding this problem, if your CK server has a relatively large memory configuration, you will not encounter it. My CK server memory is only 32G, so you may encounter this memory problem. The simplest is actually to expand the memory, but in order to avoid Memory expansion is not easy on some projects, so you need to think of other ways to solve it.

   • The first thing I thought of was to use the swap partition, but in actual testing, ClickHouse only uses physical memory, not virtual memory.
   • The second method is to where condition + paging to reduce the peak memory usage. The actual measurement is effective, but it is more troublesome.
4. Then I wondered if there are any parameters that can control the threshold of this batch write, so that I won't encounter the problem of insufficient memory, 100w will exceed the memory limit, and 50w should not exceed it. But I googled it and didn't find the corresponding parameter. No one in the group knew about it. I had no choice but to look in the source code to see if the value was hard-coded.

Three. Print stack frame

First, we need to print the stack information through pstack, otherwise we cannot know where the function entry is. Before that, we need to additionally install clickhouse-common-static-dbg , otherwise the stack information will be relatively rudimentary, and GDB debugging will follow There will also be problems.

For example, my ClickHouse version is 20.8.12.2, then the corresponding rpm package is clickhouse-common-static-dbg-20.8.12.2-2.x86_64.rpm

After executing the insert statement in ClickHouse, import it into a log file pstack + clickhouse process pid> /opt/ck_pstack.log.

4. GDB debugging

GDB does not introduce much, but personally prefer to use CGDB , just install it with Yum, the OS version I use is CentOS7.9.

Before using GDB to debug, you also need to download the source code of ClickHouse and unzip it to the /build/ directory (the default compilation directory).

Then the debugging steps are roughly:

1. First create a new window, clickhouse-client connects to ClickHouse, and waits for SQL execution.
2. Open CGDB, attach to the pid of Clickhouse, and place a breakpoint on the corresponding function line. Here, DB::SourceFromInputStream::generate (selected from the stack frame) is selected. CGDB needs to be configured to ignore the semaphore, otherwise CGDB will always be disconnected.

(gdb) att 1446
(gdb) handle SIGUSR2 noprint nostop
Signal        Stop      Print   Pass to program Description
SIGUSR2       No        No      Yes             User defined signal 2
(gdb) handle SIGUSR1 noprint nostop
Signal        Stop      Print   Pass to program Description
SIGUSR1       No        No      Yes             User defined signal 1
(gdb) b DB::SourceFromInputStream::generate
Breakpoint 1 at 0x16610f40: file ../src/Processors/Sources/SourceFromInputStream.cpp, line 135.

3. Execute the insert statement in the window opened in the first step.
4. Press c to continue in CGDB, and it will jump to the generate function

5. The next step is to slowly n, print the parameters, and look at the code flow step by step.

Five. max_block_size

Here is the result of debugging directly. When the number of rows read is equal to max_block_size, it will jump out of the loop reading, write to ClickHouse in batches, and release the memory. The GDB print value of this 1048545 is 0612f411c27d18, which looks very much like one Configurable parameters.

min_insert_block_size_rows source code. The following paragraph looks more like, determined by the 0612f411c27e4d configuration parameter.

Then I searched the system.settings table and found that the description of the min_insert_block_size_rows parameter is indeed very similar to the default value. Basically determining this parameter will affect the number of rows written in batches.

Six. Testing

Modify the parameter to 1w in the session, and then execute insert, you can run through, and no error will be reported.

localhost :) set min_insert_block_size_rows = 10000;

0 rows in set. Elapsed: 0.001 sec. 

localhost :) insert into `test`.`memory_test`  select * from mysql('192.168.213.222:3306','test','memory_test','root','xxxx');                                                  

INSERT INTO test.memory_test SELECT 
    *
FROM mysql('192.168.213.222:3306', 'test', 'memory_test', 'root', 'xxxx')
Ok.

0 rows in set. Elapsed: 2065.189 sec. Processed 500.00 thousand rows, 51.23 GB (242.11 rows/s., 24.81 MB/s.) 

Show processlist can also see that it is indeed 1w lines to write, then there will be no more memory shortage problem, so far this problem has been basically solved.