Remember a high CPU troubleshooting process

The problem

Suddenly, after deploying a very simple new business last week, it didn’t take long to go online and suddenly OOM, most of the interface access timed out, and some even failed directly. At first I thought it was caused by big data queries, but I looked at the CPU. 300%.

Troubleshooting ideas

At first I looked at the log, as follows:

org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
### The error may exist in file [/Users/bingfeng/Documents/mochuCode/service-ecook/target/classes/mybatis/mapper/CollectionSortMapper.xml]
### The error may involve cn.ecook.core.mapper.CollectionSortMapper.selectTopSortCollectionWithDays
### The error occurred while handling results
### SQL: select sortid as id ,count(*) as `number` from `collection_sort_collection`          WHERE  createtime between concat(?, ' 00:00:00') and concat(?, ' 23:59:59')          group by sortid         order by `number` DESC LIMIT ?
### Cause: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
    at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:92)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
    at com.sun.proxy.$Proxy97.selectList(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:223)
    at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
    at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
    at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:93)
    at com.sun.proxy.$Proxy166.selectTopSortCollectionWithDays(Unknown Source)

The log only reported this kind of error at the time. At first, I thought it was a simple index out-of-bounds exception. I didn't care, and I passed this problem.

Analyze processes with high CPU

At this point, we need to analyze the stack information to see where the problem is at the bottom. Here are the specific steps:

• According to the process number query to view the resource utilization rate of each thread in the process

top -Hp 2159

The CPU here is after recovery, and the result at that time is 99% for the first three CPUs.

• Convert thread id to hexadecimal

  printf "%x\n" 2205

• Print stack information

  jstack 2159 | grep -10 89d

Through the following status, it can be found that the reason for the slow interface access is because all threads are blocked, and then you can find that the cause of these problems is caused by querying SQL. Then let's take a look at the newly submitted code, where the SQL query was performed.

identify the problem

Through the investigation, it was found that there is such a piece of code, which is the entity of the SQL query result. It is because the @Builder annotation is used and there is no explicit structure provided that the CPU has been soaring.

After using Builder again, you must explicitly declare the construction method

Reflection

I think there are two reasons for this problem:

• 1. The self-test is not in place;
• 2. Insufficient specifications (if the specifications are in place, the entity class cannot be unstructured);