The strongest code self-test method in history, no one!

Special note: This is really not a title party. I have been writing code for 20+ years, and I really think go fuzzing is the most awesome code self-testing method I have ever seen. When I am using the AC自动机 algorithm to improve the keyword filtering efficiency (increase ~50%), and improve the processing mechanism of mapreduce to panic , all pass go fuzzing Found edge case bug. So I deeply think that this is the most powerful code self-test method I have ever seen, no one!

go fuzzing High quality code has been found so far Go Standard library has more than 200 bugs, see: https://github.com/dvyukov/go-fuzz#trophies

The blessings among programmers during the Spring Festival are often, I wish you no bugs in your code! Although it's a joke, it's a fact that we write bugs every day for each of us programmers. The fact that the code has no bugs can only be falsified, not proven. The upcoming Go 1.18 official provides a great tool to help us falsify - go fuzzing .

Go 1.18 everyone is most concerned about generics, but I really think go fuzzing is really the most useful feature of Go 1.18, no one!

Let's take a look at this article in detail go fuzzing：

• what is it?
• how to use?
• What are the best practices?

First, you need to upgrade to Go 1.18

Go 1.18 is not yet officially released, but you can download the RC version, and even if you use an earlier version of Go in production, you can use go fuzzing in the development environment to find bugs

what is go fuzzing

According to the official documentation , go fuzzing is to automate testing by continuously giving a program different inputs, and intelligently find failed cases by analyzing code coverage. This method can find some edge cases as much as possible, and the pro-test did find some problems that are usually difficult to find.

how to use go fuzzing

Officially introduce some rules for writing fuzz tests:

• The function must start with Fuzz, the only parameter is *testing.F , there is no return value
• Fuzz tests must be in the file *_test.go
• The fuzz target in the above picture is a method call (*testing.F).Fuzz , the first parameter is *testing.T , and then the parameter called fuzzing arguments , c29 no return value
• There can only be one in each fuzz test fuzz target
• When calling f.Add(…) , the parameter type needs to be the same as fuzzing arguments in the same order and type

• fuzzing arguments Only the following types are supported:

  • string , []byte
  • int , int8 , int16 , int32 / rune , int64
  • uint , uint8 / byte , uint16 , uint32 , uint64
  • float32 , float64
  • bool
• fuzz target Don't rely on global state, it will run in parallel.

run fuzzing tests

If I write a fuzzing test like:

 // 具体代码见 https://github.com/zeromicro/go-zero/blob/master/core/mr/mapreduce_fuzz_test.go
func FuzzMapReduce(f *testing.F) {
  ...
}

Then we can do it like this:

 go test -fuzz=MapReduce

We will get something like this:

 fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 3s, execs: 3338 (1112/sec), new interesting: 56 (total: 57)
fuzz: elapsed: 6s, execs: 6770 (1144/sec), new interesting: 62 (total: 63)
fuzz: elapsed: 9s, execs: 10157 (1129/sec), new interesting: 69 (total: 70)
fuzz: elapsed: 12s, execs: 13586 (1143/sec), new interesting: 72 (total: 73)
^Cfuzz: elapsed: 13s, execs: 14031 (1084/sec), new interesting: 72 (total: 73)
PASS
ok    github.com/zeromicro/go-zero/core/mr  13.169s

Among them ^C I pressed ctrl-C to terminate the test, please refer to the official documentation for detailed explanation.

Best practices for go-zero

According to the experience I have used, I initially summarize the best practice into the following four steps:

1. Define fuzzing arguments , first of all, we need to understand how to define fuzzing arguments , and write ---646b3864---d8edc1a578f by the given fuzzing arguments fuzzing target
2. Thinking about how to write fuzzing target , the focus here is how to verify the correctness of the result, because fuzzing arguments is given by "random", so there must be a general result verification method
3. Think about how to print the results in case of failure, so as to generate new ones unit test
4. 根据失败的fuzzing test a42682a8f9007603fda291df0d24d49f---打印结果编写新的unit test，这个新的 unit test 会被用来调试解决 fuzzing test 发现的问题，并固化下来留给 CI 用

Next, we will show the above steps with the simplest array summation function. The actual case of go-zero is slightly complicated. At the end of the article, I will give the internal landing case of go-zero for your reference to write complex scenarios.

Here is a code implementation of a bug-injected summation:

 func Sum(vals []int64) int64 {
  var total int64

  for _, val := range vals {
    if val%1e5 != 0 {
      total += val
    }
  }

  return total
}

1. Definition fuzzing arguments

You need to give at least one fuzzing argument , otherwise go fuzzing can't generate test code, so even if we don't have good input, we need to define a fuzzing argument ，这里我们就用slice 元素fuzzing arguments ， Go fuzzing跑出来的code coverage参数来模拟测试.

 func FuzzSum(f *testing.F) {
  f.Add(10)
  f.Fuzz(func(t *testing.T, n int) {
    n %= 20
    ...
  })
}

Here n is to let go fuzzing simulate the number of slice elements. In order to ensure that the number of elements is not too many, we limit it to less than 20 (0 is no problem), and we A corpus with a value of 10 is added ( go fuzzing which is called corpus ), this value is a value that makes go fuzzing a cold start. important.

2. How to write fuzzing target

The focus of this step is how to write verifiable fuzzing target , according to the given fuzzing arguments writing the test code, it is also necessary to generate data for verifying the correctness of the result.

For our Sum function, it is actually relatively simple, that is, randomly generate a slice of n elements, and then sum to calculate the desired result. as follows:

 func FuzzSum(f *testing.F) {
  rand.Seed(time.Now().UnixNano())

  f.Add(10)
  f.Fuzz(func(t *testing.T, n int) {
    n %= 20
    var vals []int64
    var expect int64
    for i := 0; i < n; i++ {
      val := rand.Int63() % 1e6
      vals = append(vals, val)
      expect += val
    }

    assert.Equal(t, expect, Sum(vals))
  })
}

This code is still very easy to understand, the summation by yourself Sum is just for comparison, so I won't explain it in detail. But in complex scenarios, you need to think carefully about how to write verification code, but this is not too difficult. If it is too difficult, it may be that you do not have enough understanding or simplification of the test function.

At this point, you can use the following command to run fuzzing tests , and the result is similar to the following:

 $ go test -fuzz=Sum
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 0s, execs: 6672 (33646/sec), new interesting: 7 (total: 6)
--- FAIL: FuzzSum (0.21s)
    --- FAIL: FuzzSum (0.00s)
        sum_fuzz_test.go:34:
              Error Trace:  sum_fuzz_test.go:34
                                  value.go:556
                                  value.go:339
                                  fuzz.go:334
              Error:        Not equal:
                            expected: 8736932
                            actual  : 8636932
              Test:         FuzzSum

    Failing input written to testdata/fuzz/FuzzSum/739002313aceff0ff5ef993030bbde9115541cabee2554e6c9f3faaf581f2004
    To re-run:
    go test -run=FuzzSum/739002313aceff0ff5ef993030bbde9115541cabee2554e6c9f3faaf581f2004
FAIL
exit status 1
FAIL  github.com/kevwan/fuzzing  0.614s

Then here comes the problem! We have seen that the result is wrong, but it is difficult for us to analyze why it is wrong. If you look carefully, how do you analyze the above output?

3. How to print input in failure case

For the failed test above, if we can print out the input and form a simple test case, then we can debug it directly. It is best to print the input directly copy/paste to the new test case, if the format is not correct, for so many lines of input, you need to adjust the format line by line, and it is not necessarily only one Failed case.

So we changed the code to the following:

 func FuzzSum(f *testing.F) {
  rand.Seed(time.Now().UnixNano())

  f.Add(10)
  f.Fuzz(func(t *testing.T, n int) {
    n %= 20
    var vals []int64
    var expect int64
    var buf strings.Builder
    buf.WriteString("\n")
    for i := 0; i < n; i++ {
      val := rand.Int63() % 1e6
      vals = append(vals, val)
      expect += val
      buf.WriteString(fmt.Sprintf("%d,\n", val))
    }

    assert.Equal(t, expect, Sum(vals), buf.String())
  })
}

Run the command again and get the following result:

 $ go test -fuzz=Sum
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 0s, execs: 1402 (10028/sec), new interesting: 10 (total: 8)
--- FAIL: FuzzSum (0.16s)
    --- FAIL: FuzzSum (0.00s)
        sum_fuzz_test.go:34:
              Error Trace:  sum_fuzz_test.go:34
                                  value.go:556
                                  value.go:339
                                  fuzz.go:334
              Error:        Not equal:
                            expected: 5823336
                            actual  : 5623336
              Test:         FuzzSum
              Messages:
                            799023,
                            110387,
                            811082,
                            115543,
                            859422,
                            997646,
                            200000,
                            399008,
                            7905,
                            931332,
                            591988,

    Failing input written to testdata/fuzz/FuzzSum/26d024acf85aae88f3291bf7e1c6f473eab8b051f2adb1bf05d4491bc49f5767
    To re-run:
    go test -run=FuzzSum/26d024acf85aae88f3291bf7e1c6f473eab8b051f2adb1bf05d4491bc49f5767
FAIL
exit status 1
FAIL  github.com/kevwan/fuzzing  0.602s

4. Write new test cases

According to the output of the failure case above, we can copy/paste generate the following code. Of course, the framework is written by itself, and the input parameters can be copied directly.

 func TestSumFuzzCase1(t *testing.T) {
  vals := []int64{
    799023,
    110387,
    811082,
    115543,
    859422,
    997646,
    200000,
    399008,
    7905,
    931332,
    591988,
  }
  assert.Equal(t, int64(5823336), Sum(vals))
}

In this way, we can easily debug, and can add a valid unit test to ensure that this bug will never appear again.

go fuzzing more experience

Go version issue

I believe that Go 1.18 is released, and most of the online code of the project will not be upgraded to 1.18 immediately, so what should I do if the go fuzzing introduced testing.F cannot be used?

Online (go.mod) does not upgrade to Go 1.18, but our local machine is fully recommended to upgrade, then we only need to put the above FuzzSum into a file with a similar name sum_fuzz_test.go file, and then add the following command to the file header:

 //go:build go1.18
// +build go1.18

Note: The third line must be a blank line, otherwise it will become a comment of package .

In this way, no matter which version we use online, we will not report an error, and we run fuzz testing which is generally run locally and is not affected.

go fuzzing unreproducible failures

The steps mentioned above are for simple cases, but sometimes a new one unit test is formed based on the input obtained from the failed case and the problem cannot be reproduced (especially if there is a goroutine deadlock problem), the problem becomes complicated. Now, you can feel the following output:

 go test -fuzz=MapReduce
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 10 workers
fuzz: elapsed: 3s, execs: 3681 (1227/sec), new interesting: 54 (total: 55)
...
fuzz: elapsed: 1m21s, execs: 92705 (1101/sec), new interesting: 85 (total: 86)
--- FAIL: FuzzMapReduce (80.96s)
    fuzzing process hung or terminated unexpectedly: exit status 2
    Failing input written to testdata/fuzz/FuzzMapReduce/ee6a61e8c968adad2e629fba11984532cac5d177c4899d3e0b7c2949a0a3d840
    To re-run:
    go test -run=FuzzMapReduce/ee6a61e8c968adad2e629fba11984532cac5d177c4899d3e0b7c2949a0a3d840
FAIL
exit status 1
FAIL  github.com/zeromicro/go-zero/core/mr  81.471s

In this case, just tell us fuzzing process stuck or ended abnormally, and the status code is 2. In this case, generally re-run will not reproduce. Why simply return error code 2? I looked carefully at the source code of go fuzzing , each fuzzing test is run by a separate process, and then go fuzzing throws away the process output of the fuzzing test , just showing the status code. So how do we solve this problem?

After careful analysis, I decided to write a regular unit test code similar to fuzzing test , so that the failure can be guaranteed to be in the same process, and the error message will be printed to the standard output, the code is roughly as follows :

 func TestSumFuzzRandom(t *testing.T) {
  const times = 100000
  rand.Seed(time.Now().UnixNano())

  for i := 0; i < times; i++ {
    n := rand.Intn(20)
    var vals []int64
    var expect int64
    var buf strings.Builder
    buf.WriteString("\n")
    for i := 0; i < n; i++ {
      val := rand.Int63() % 1e6
      vals = append(vals, val)
      expect += val
      buf.WriteString(fmt.Sprintf("%d,\n", val))
    }

    assert.Equal(t, expect, Sum(vals), buf.String())
  }
}

This way we can simply simulate it go fuzzing , but we can get clear output for any errors. Maybe I didn't study it thoroughly here go fuzzing , or there are other ways to control it, if you know, thank you for letting me know.

But this kind of simulation case that takes a long time to run, we don't want it to be executed every time during CI, so I put it in a separate file with a file name like sum_fuzzcase_test.go , and Add the following command to the file header:

 //go:build fuzz
// +build fuzz

In this way, we need to add -tags fuzz when running this simulation case, for example:

 go test -tags fuzz ./...

Complex usage example

The above is an example, which is relatively simple. If you don't know how to write a complex scene, you can first see how go-zero landed go fuzzing , as shown below:

• MapReduce - https://github.com/zeromicro/go-zero/tree/master/core/mr

  • Fuzzy tested deadlock and goroutine leak , especially the complex scenarios of chan + goroutine can be used for reference

• stringx - https://github.com/zeromicro/go-zero/tree/master/core/stringx

  • Fuzzing tests the implementation of conventional algorithms, which can be used for reference in algorithmic scenarios

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.