Redigo: Analysis of ScanStruct() anonymous pointer field

Redigo issue 487

How to scan struct with nested fields?#487
In order to better understand this article, it is recommended to read the original issue first

1. What is the problem

Parse the data returned by the HGETALL command into the corresponding structure UserInfo , but the data in the *LiteUser field in the structure cannot be successfully parsed.

If you change *LiteUser to LiteUser it will work.

2. Reproduce the problem

1. Copy the code in the issue, install redigo in go module, and prepare a redis service.
2. The version in redigo in go.mod is set to the version before the issue is unmodified: v1.8.1.
3. Running the code will reproduce the problem of this issue. The data of the *LiteUser field cannot be successfully parsed.

Try the latest version of the code and run it:

1. The version in redigo in go.mod is set to the latest version of issue: v1.8.8.
2. Running the code, the problem does not appear.

Note that in order to reproduce the problem in the latest version, you need to add the following code below about line 73 of the sample code (we will come back to this problem later):

 ...
var newUser UserInfo
newUser.LiteUser = &LiteUser{}
...

Third, how to solve

For details, see pr 490

Before looking at how to solve it, let's sort out the execution process:

3.1 Parsing data into structure variables

When executing HGETALL and getting the data from Redis, it needs to parse the data into the member variables of the structure, just like taking the data from MySQL, and parsing it into the member variables of the structure means a meaning.

Redigo provides a method to pass data and structure variables in, and the data will be parsed to the newUser structure:

 redis.ScanStruct(v, &newUser)

3.2 ScanStruct

Next, see what redis.ScanStruct() has done.
I combed and summarized the methods called in the process:

 // 将数据解析到structSpecForType返回的结构体成员上
func ScanStruct(src []interface{}, dest interface{}) error {
    //获取变量指针
    d := reflect.ValueOf(dest)
    //获取指针指向的变量
    d = d.Elem()
    structSpecForType(d.Type())
    ...
}

// 根据传入的reflect.Type，先去缓存中查找是否解析过，如果没有调用compileStructSpec
func structSpecForType(t reflect.Type) *structSpec {
    ...
    compileStructSpec(t, make(map[string]int), nil, ss)
    ...
}

3.3 compileStructSpec

compileStructSpec method implements type resolution, and the problem lies here.

First post the summarized summary:

• Use reflection to parse data to all member variables of &newUser 结构体
• In V1.8.1 and earlier versions, only reflect.Struct (LiteUser) was parsed, and reflect.Ptr (*LiteUser) was not processed
• In V1.8.2 and later, the judgment of reflect.Ptr is added

Here is the core logic before the fix:

 func compileStructSpec(t reflect.Type, depth map[string]int, index []int, ss *structSpec) {
   // t.NumField()获取结构体类型的所有字段的个数
   for i := 0; i < t.NumField(); i++ {
      // t.Field()返回指定的字段，类型为 StructField
      f := t.Field(i)
      switch {
      // f.PkgPath 包路径不为空 且 不是匿名函数
      // f.Anonymous 表示该字段是否为匿名字段
      case f.PkgPath != "" && !f.Anonymous:
         // 忽略未导出的：结构体中的某个成员改为小写（私有），就会进到这个case
         // Ignore unexported fields.
      // UserInfo中的成员LiteUser，并未设置 name，为匿名字段，就会进到这个case
      case f.Anonymous:
         // f.Type.Kind() 获取种类
        // 如果当前type为结构体，进行递归调用，以处理当前type内所有结构体成员
        // 对于 `LiteUser` 会进到这个 case
         if f.Type.Kind() == reflect.Struct {
            compileStructSpec(f.Type, depth, append(index, i), ss)
         }

After fix:

 ...
func compileStructSpec(t reflect.Type, depth map[string]int, index []int, ss *structSpec) {
LOOP:
   for i := 0; i < t.NumField(); i++ {
      f := t.Field(i)
      switch {
      case f.PkgPath != "" && !f.Anonymous:
         // Ignore unexported fields.
      case f.Anonymous:
         switch f.Type.Kind() {
         case reflect.Struct:
            compileStructSpec(f.Type, depth, append(index, i), ss)
         // 这里是变动的部分，对于 `*LiteUser` 会进到这个 case
         case reflect.Ptr:
            // 如果当前字段的type的值为结构体，进行递归调用，以处理当前字段内所有结构体成员
            // f.Type.Kind()返回的是前f的种类，也就是reflect.Ptr
            // f.Type.Elem().Kind() 返回的是前f的值的种类，也就是reflect.Struct
            // TODO(steve): Protect against infinite recursion.
            if f.Type.Elem().Kind() == reflect.Struct {
               compileStructSpec(f.Type.Elem(), depth, append(index, i), ss)
            }
         }
...

OK~, problem solved!

4. Expansion

4.1 Reflection

compileStructSpec Inside the method, it is mainly realized by reflection.
The key point here is to talk about why d := reflect.ValueOf(dest) After finishing, I still need to use d = d.Elem() , citing a sentence from "Go Language Design and Implementation"

Since function calls in Go language are all passed by value, we can only change the original variable in a roundabout way: first get the corresponding reflect.Value of the pointer, and then get it through the reflect.Value.Elem method Variables that can be set.

refer to
Go language involves and implements - reflection

4.2 newUser.LiteUser = &LiteUser{}

If int, string, etc. are value types, even if they are not initialized, but only declared, the value will default to the "zero" value of this type.
But reference variables like Map, Slice, Channel, etc. need to be make() before use.

Similarly, a variable of type & whose value is stored in a memory address must first initialize a structure of LiteUser , and then assign its memory address to newUser.LiteUser can be used normally.