Finally solved this online panic problem

From the public Gopher refers to the north

I don't know if other people are like this. Anyway, the word that Lao Xu is most afraid of hearing is "occasionally". As for the reason, I won't say much.

Look directly at the panic information below.

runtime error: invalid memory address or nil pointer dereference

panic(0xbd1c80, 0x1271710)
        /root/.go/src/runtime/panic.go:969 +0x175
github.com/json-iterator/go.(*Stream).WriteStringWithHTMLEscaped(0xc00b0c6000, 0x0, 0x24)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/stream_str.go:227 +0x7b
github.com/json-iterator/go.(*htmlEscapedStringEncoder).Encode(0x12b9250, 0xc0096c4c00, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/config.go:263 +0x45
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc002e9c8d0, 0xc0096c4c00, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:110 +0x78
github.com/json-iterator/go.(*structEncoder).Encode(0xc002e9c9c0, 0xc0096c4c00, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:158 +0x3f4
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc002eac990, 0xc0096c4c00, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:110 +0x78
github.com/json-iterator/go.(*structEncoder).Encode(0xc002eacba0, 0xc0096c4c00, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:158 +0x3f4
github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc002e9f570, 0xc006b18b38, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect_optional.go:70 +0xf4
github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc002e9f580, 0xc0096c4c00, 0xc00b0c6000)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect.go:219 +0x68
github.com/json-iterator/go.(*Stream).WriteVal(0xc00b0c6000, 0xb78d60, 0xc0096c4c00)
        /go/pkg/mod/github.com/json-iterator/go@v1.1.11/reflect.go:98 +0x150
github.com/json-iterator/go.(*frozenConfig).Marshal(0xc00012c640, 0xb78d60, 0xc0096c4c00, 0x0, 0x0, 0x0, 0x0, 0x0)

First of all, I firmly believe that the power of open source is trustworthy. Therefore, the first wave of operation of Lao Xu is to analyze whether there are logical loopholes in the business code. It is clear that colleagues are also trustworthy, so it is decisive to guess that some unanticipated data triggers the boundary conditions. The next step is to save the routine operation of the scene.

As the title says, this is an occasional panic problem, so you can save the scene according to the above classification in a method that conforms to the current technology stack. Next comes the season of waiting for the harvest, and the wait is many days. I received an alarm several times in the middle, but the scene did not meet the expectations.

At this time, not only did I not panic, but I was even a little excited. So-and-so once said: "You must dare to question, dare to challenge authority." After thinking about it, you can't control it. I have to make a contribution to the open source cause again! json-iterator to do what you say, and start reading the source code of 061daea736a976 with careful thought.

As soon as I started to study, I realized that the saying "When God closes this door, he will open another door for you" is a lie. Lao Xu only felt that God not only closed all the doors and even closed all the windows. Let's see how he closes the door.

func (cfg *frozenConfig) Marshal(v interface{}) ([]byte, error) {
    stream := cfg.BorrowStream(nil)
    defer cfg.ReturnStream(stream)
    stream.WriteVal(v)
    if stream.Error != nil {
        return nil, stream.Error
    }
    result := stream.Buffer()
    copied := make([]byte, len(result))
    copy(copied, result)
    return copied, nil
}


// WriteVal copy the go interface into underlying JSON, same as json.Marshal
func (stream *Stream) WriteVal(val interface{}) {
    if nil == val {
        stream.WriteNil()
        return
    }
    // 省略其他代码
}

According to the panic stack, it is known that the panic is caused by the null pointer, and the non-null judgment has been made inside the (*frozenConfig).Marshal At this point, Lao Xu really had no choice but to give up strategically to solve this panic. After all, the impact is not that big, and there is no bug that programmers can fix. After such a comfort, my heart is indeed much easier to accept.

In fact, I have consciously ignored this problem for a long time, after all, I have not found the root cause of the problem. This question continued on the line until a day when I couldn't say anything, in short, the interest came, and I took two eyes again, and these two eyes are very important!

func doReq() {
    req := paramsPool.Get().(*model.Params)
    // defer 1
    defer func() {
        reqBytes, _ := json.Marshal(req)
        // 省略其他打印日志的代码
    }()
    // defer 2
    defer paramsPool.Put(req)
    // req初始化以及发起请求和其他操作
}

Note:

1. The above code variable naming has been generalized by Lao Xu.
2. The actual code in the project is far more complex than the above, but the above code is still the smallest prototype that caused this problem.

In the above code, paramsPool is a variable of type sync.Pool sync.Pool must be familiar to everyone. sync.Pool is to reuse objects that have been used (coroutine safety), reduce memory allocation and reduce GC pressure.

type test struct {
    a string
}

var sp = sync.Pool{
    New: func() interface{} {
        return new(test)
    },
}

func main() {
    t := sp.Get().(*test)
    fmt.Println(unsafe.Pointer(t))
    sp.Put(t)
    t1 := sp.Get().(*test)
    t2 := sp.Get().(*test)
    fmt.Println(unsafe.Pointer(t1), unsafe.Pointer(t2))
}

According to the above code and output results, the t1 variable and t variable address are the same, so they are multiplexed objects. At this point, it is easy to find the root cause of the problem by doReq

defer 2 order of 061daea736ab2b and defer 1 ! !

defer 2 order of 061daea736ab3d and defer 1 ! !

defer 2 order of 061daea736ab4f and defer 1 ! !

sync.Pool provided Get and Put methods coroutine is safe, but the high concurrent calls doReq a function json.Marshal(req) and initialization request concurrency problems, very likely to cause panic concurrent calls timeline shown below.

Now that the cause has been found, it is much easier to solve, just adjust the calling order of defer 2 and defer 1 After Lao Xu posted the modified code online, there was no panic again. The root cause of this accident is a trivial detail, so we usually have to be cautious and cautious in development to avoid irreparable losses caused by such small mistakes. Another rule of thumb is to try not to get into the nitty-gritty when developing and checking problems. Proper pauses may have unexpected and miraculous effects.

Finally, I sincerely hope that this article can be helpful to all readers.