Application and sharing of gjson in GO

[TOC]

Application and sharing of `gjson`

Last time we shared the data of using GO to crawl static web pages, let’s review it together

Brief description of sharing static and dynamic pages
GO crawls simple data of static web pages
GO crawls pictures on the web
Crawl resources on the web page concurrently

If you are interested in GO crawling static data, please check the article share a wave of GO crawlers

`json` is 06104fc058cc69?

JSON (JavaScript Object Notation, JS object notation) is a lightweight data exchange format
JS specification formulated by the European Computer Association), and uses a text format completely independent of the programming language to store and represent data

json has the following advantages:

Concise and clear hierarchy
Easy to read and write
Easy to parse and generate by machine
Can improve network transmission efficiency

Briefly list our commonly used data serialization methods

json
xml

It is an extensible markup language, a simple data storage language

protobuf

It is a platform-independent, language-independent, scalable, portable and efficient serialized data structure protocol , which can be used for network communication and data storage

`gjson` is 06104fc058ceb9?

is a library in GO

It mainly provides a very fast and simple way to get the corresponding value from the json

The gjson library is get + json the abbreviation of 06104fc058cf22. Coincidentally, there is also the sjson library. Friends will know what it means. It means set + json

How to use `gjson`

Regarding gjson , XDM, I am here to sort out the basic use of this library, the knowledge points involved, and the matters needing attention.

If you want to see gjson the source code of 06104fc058cf6e achieves efficient and fast operation of json , interested friends, you can check the source code in detail based on the first reference.

The sharing of this article focuses on the following 4 aspects to practice and sort out the use of gjson

Simple use of gjson
gjson of json line
Modifiers and custom modifiers of gjson
gjson key path matching rules

Simple use of `gjson`

Let's simply use a gjson , the following coding involves the following points:

Set specific json data
checks whether the json data
Get a single value at once
Get multiple values at once

package main

import (
   "log"
   "github.com/tidwall/gjson"
)

func main() {

   // 设置参数，打印行数
   log.SetFlags(log.Lshortfile | log.LstdFlags)
   
  // 设置 json 数据
   json := `
         {
            "author": {
               "name": "xiaomotong",
               "age": 18,
               "hobby": "writing"
            },
            "extra": "hello wolrd"
            "picList":[{"name":"xiaozhu1"},{"name":"xiaozhu2"}]
         }
         `
   // 校验 json 字符串是否合法
   // 如果不合法的话， gjson 不会报错 panic，可能会拿到一个奇怪值
   if gjson.Valid(json){
      log.Println("json valid ...")
   }else{
      log.Fatal("json invalid ... ")
   }

   // 获取 author.name 的 值
   aName := gjson.Get(json, "author.name")
   log.Println("aName :", aName.String())

   // 获取 extra 的值
   extra := gjson.Get(json, "extra")
   log.Println("extra:", extra)

   // 获取 一个不存在的 键 对应的 值
   non := gjson.Get(json, "non")
   log.Println("non:", non)


   // 一次性 获取json 的多个键 值
    res := gjson.GetMany(json, "author.age", "author.hobby","picList")
    for i, v := range res{
        if i == 0{
            log.Println(v.Int())
        }else if i == 2{
            for _,vv := range v.Array(){
                log.Println("picList.name :",vv.Get("name"))
            }
        }else{
            log.Println(v)
        }
    }
}

After running the above code, you can see the following effects:

2021/06/xx xx:32:04 main.go:28: json valid ...
2021/06/xx xx:32:04 main.go:35: aName : xiaomotong
2021/06/xx xx:32:04 main.go:39: extra: hello wolrd
2021/06/xx xx:32:04 main.go:43: non:
2021/06/xx xx:32:04 main.go:50: 18
2021/06/xx xx:32:04 main.go:57: writing
2021/06/xx xx:32:04 main.go:53: picList.name : xiaozhu1
2021/06/xx xx:32:04 main.go:53: picList.name : xiaozhu2

We need to pay attention to getting our data source right, that is, our json data must be legal, otherwise, gjson library will not be the value we expect

Use gjson.Get() to get a single value
Use gjson.GetMany() to get multiple values
Use gjson.Valid() to determine json data of 06104fc058d2a2 is valid

`gjson` of `json`

Let's take a look at the json line

gjson provides the following syntax to parse json row data:

..#

Output the length of the json

..#.author

Output the value corresponding to json each row of author

..#(author="xiaomotong").hobby
In the output json line, the hobby value corresponding to the line where author = xiaomotong
..1

Output the json line of the 06104fc058d42b line array, if ..2 , output the 3 line

Traverse json line

Use gjson.ForEachLine traverse json row, and the details in each row of data can also be traversed

Let's write a DEMO to cover the syntax that needs to be used above:

package main

import (
   "github.com/tidwall/gjson"
   "log"
)

const json = `
   {"author": "xiaomotong", "age": 18, "hobby":"play"}
   {"author": "xiaozhu", "age": 19 , "hobby":"eat"}
   {"author": "zhangsan", "age": 20, "hobby":"drink"}
   {"author": "lisi", "age": 21, "hobby":"sleep"}`

func main() {

   // 设置参数，打印行数
   log.SetFlags(log.Lshortfile | log.LstdFlags)

   // 输出 json 行数组的长度
   log.Println(gjson.Get(json, "..#"))
    
   // 输出 json 行 数组的第 3 行
   log.Println(gjson.Get(json, "..2"))
    
   // 输出 json 每一行 里面的 author 对应的值，组成一个数组
   log.Println(gjson.Get(json, "..#.author"))
    
   // 输出输出 json 行 中，author = xiaomotong 所在行 对应的 hobby 值
   log.Println(gjson.Get(json, `..#(author="xiaomotong").hobby`))

   // 遍历 json 行
   gjson.ForEachLine(json, func(jLine gjson.Result) bool {
      log.Println("author:", gjson.Get(jLine.String(), "hobby"))
      return true
   })
}

After the above code runs, the result is as follows:

2021/06/xx xx:17:52 main2.go:20: 4
2021/06/xx xx:17:52 main2.go:22: {"author": "zhangsan", "age": 20, "hobby":"drink"}
2021/06/xx xx:17:52 main2.go:24: ["xiaomotong","xiaozhu","zhangsan","lisi"]
2021/06/xx xx:17:52 main2.go:26: play
2021/06/xx xx:17:52 main2.go:30: author: play
2021/06/xx xx:17:52 main2.go:30: author: eat
2021/06/xx xx:17:52 main2.go:30: author: drink
2021/06/xx xx:17:52 main2.go:30: author: sleep

Let's take a look at the implementation of gjson.ForEachLine

// ForEachLine iterates through lines of JSON as specified by the JSON Lines
// format (http://jsonlines.org/).
// Each line is returned as a GJSON Result.
func ForEachLine(json string, iterator func(line Result) bool) {
   var res Result
   var i int
   for {
      i, res, _ = parseAny(json, i, true)
      if !res.Exists() {
         break
      }
      if !iterator(res) {
         return
      }
   }
}

Each row will return a JSON results, Result

parseAny is the specific json data that parseAny function will involve in detail how to judge and process each character

// parseAny parses the next value from a json string.
// A Result is returned when the hit param is set.
// The return values are (i int, res Result, ok bool)
func parseAny(json string, i int, hit bool) (int, Result, bool) {
   var res Result
   var val string
   // 一个字符一个字符的做处理
   // 不同的字符 有对应的逻辑，感兴趣的XDM 可以细品
   for ; i < len(json); i++ {
      if json[i] == '{' || json[i] == '[' {
         i, val = parseSquash(json, i)
         if hit {
             // 对应字符赋值
            res.Raw = val
            res.Type = JSON
         }
         return i, res, true
      }
      if json[i] <= ' ' {
         continue
      }
      // 排除上述特殊几种情况后，继续按照下述情况进行处理字符
      switch json[i] {
      case '"':
         i++
         var vesc bool
         var ok bool
         i, val, vesc, ok = parseString(json, i)
         if !ok {
            return i, res, false
         }
         if hit {
            res.Type = String
            res.Raw = val
            if vesc {
               res.Str = unescape(val[1 : len(val)-1])
            } else {
               res.Str = val[1 : len(val)-1]
            }
         }
         return i, res, true
      case '-', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
         i, val = parseNumber(json, i)
         if hit {
            res.Raw = val
            res.Type = Number
            res.Num, _ = strconv.ParseFloat(val, 64)
         }
         return i, res, true
      case 't', 'f', 'n':
         vc := json[i]
         i, val = parseLiteral(json, i)
         if hit {
            res.Raw = val
            switch vc {
            case 't':
               res.Type = True
            case 'f':
               res.Type = False
            }
            return i, res, true
         }
      }
   }
   return i, res, false
}

Let's take a look at the specific data structure of Result

// Result represents a json value that is returned from Get().
type Result struct {
   // Type is the json type
   Type Type
   // Raw is the raw json
   Raw string
   // Str is the json string
   Str string
   // Num is the json number
   Num float64
   // Index of raw value in original json, zero means index unknown
   Index int
}

According to the difference of Type Type , corresponding to the different data in Str string , Num float64 , Index int

Here’s Type let’s see what happens

const (
   // Null is a null json value
   Null Type = iota
   // False is a json false boolean
   False
   // Number is json number
   Number
   // String is a json string
   String
   // True is a json true boolean
   True
   // JSON is a raw block of JSON
   JSON
)

So it seems clear at a glance, or if gjson want to further study the corresponding characters of the gjson and see the specific implementation in the gjson.go

`gjson key path matching rules`

What is the

It is . . Let’s make a table to see which matching rules gjson

tag	instruction
?	Match a single character, for example `hell?` can match the hello key, but not helloo
*	Match any number of characters, for example `hell*` can match hello, helloooo, both
xx.xx	Used for matching arrays, for example, hello is an array, then `hello.0` is the first element of the matching array, `hello.1` is the 2 matched elements
xx.#	Get the length of the array, for example hello.#
`.` appears in the key name, then it needs to be escaped `\`	`hello.world` understand. For example, the key name is 06104fc058d81b, , when you need to use this key, you need to escape `hello\.world`
==、!=、<、<=、>、>=	For example, `hello` is a group number. The element fields in the array are `name` and `age` we match, we can add `#` to flexibly match the data we want For example: hello.#(name="xiaozhu").age
%	Pattern matching, for example hello.#(name%"n*").age
!%	Pattern matching, for example hello.#(name!%"n*").age

Let's use the above matching rules together:

package main

import (
   "github.com/tidwall/gjson"
   "log"
)
// json 源 数据
const json = `
{
  "author":{"name":"xiaomotong", "nick": "xiaozhu"},
  "age": 18,
  "hobby": ["play", "eat", "drink"],
  "love.music": "one day",
  "location": [
    {"province": "gd", "city":"gz", "area": "huangpu"},
    {"province": "gd", "city":"sz", "area": "nanshan"},
  ]
}
`

func main() {

   // 设置参数，打印行数
   log.SetFlags(log.Lshortfile | log.LstdFlags)

   // 获取名字
   log.Println("author:", gjson.Get(json, "author.name"))
   // 获取年龄
   log.Println("age:", gjson.Get(json, "age"))
    
   // 使用 ? #  *操作 hobby
   log.Println("hobby:", gjson.Get(json, "hobb?"))
   log.Println("hobby count:", gjson.Get(json, "hobby.#"))
   log.Println("second hobby:", gjson.Get(json, "ho?by.1"))
   log.Println("third hobby:", gjson.Get(json, "ho*.2"))
    
   // 键中 带有 .   我们用 \. 来转义
   log.Println("love.music", gjson.Get(json, `love\.music`))
    
   // 获取数组里面的元素 ，若我们需要获取location数组第一个元素里面的 city ，我们可以这样 gjson.Get(json, "location.0.city")
   log.Println("location first city :", gjson.Get(json, "location.0"))
   log.Println("location second city :", gjson.Get(json, "location.1"))
}

The results of the above code are as follows:

2021/06/xx xx:03:26 main3.go:27: author: xiaomotong
2021/06/xx xx:03:26 main3.go:29: age: 18
2021/06/xx xx:03:26 main3.go:31: hobby: ["play", "eat", "drink"]
2021/06/xx xx:03:26 main3.go:32: hobby count: 3
2021/06/xx xx:03:26 main3.go:34: second hobby: eat
2021/06/xx xx:03:26 main3.go:35: third hobby: drink
2021/06/xx xx:03:26 main3.go:37: love.music one day
2021/06/xx xx:03:26 main3.go:39: location first city : {"province": "gd", "city":"gz", "area": "huangpu"}
2021/06/xx xx:03:26 main3.go:40: location second city : {"province": "gd", "city":"sz", "area": "nanshan"}

gjson library are not difficult. We can take a look at the usage and find out by ourselves. The memory will be more profound. When it is really used, check it again and it will not be unfamiliar.

`Modifiers and custom modifiers of gjson`

Finally, let’s talk about gjson library. Modifiers are also very powerful. They usually play with key addresses.

Let's first sort out what are the built-in modifiers:

tag	instruction
@reverse	Flip an array
@ugly	Remove all whitespace in JSON
@valid	Verify the legitimacy of JSON
@pretty	Make JSON easier to use and read
@flatten	Array flattening, that is, ["Little Pig1", ["Little Pig2", "Little Pig3"]] will be converted to ["Little Pig1","Little Pig2","Little Pig3"]
@this	Returns the current element, can be used to return the root element
@join	Combine multiple objects into one object

Continue on DEMO

package main

import (
    "github.com/tidwall/gjson"
    "log"
)

const json = `
{
  "author":{"name":"xiaomotong", "nick": "xiaozhu"},
  "age": 18,
  "hobby": ["play", "eat", "drink"],
  "love.music": "one day",
  "location": [
    {"province": "gd", "city":"gz", "area": "huangpu"},
    {"province": "gd", "city":"sz", "area": "nanshan"},
  ]
}
`
func main() {

    // 设置参数，打印行数
    log.SetFlags(log.Lshortfile | log.LstdFlags)

    // 翻转 hobby 数组
    log.Println("翻转 hobby 数组 hobby reverse:", gjson.Get(json, "hobby|@reverse"))
    
    // 移除空白符
    log.Println("移除空白符 location.0:", gjson.Get(json, "location.0|@ugly"))

    // 使json 更加容易阅读 pretty
    log.Println("使json 更加容易阅读 pretty location : ", gjson.Get(json, "location.1|@pretty"))
    
    // 输出整个json
    log.Println(" 输出整个json this : ", gjson.Get(json, "@this"))

    test := `["小猪1", ["小猪2", "小猪3"]]`
    // 扁平化
    log.Println("扁平化 this : ", gjson.Get(test, "@flatten"))
}

Running the above code, we can see the following effects:

2021/06/xx xx:30:24 main4.go:27: 翻转 hobby 数组 hobby reverse: ["drink","eat","play"]
2021/06/xx xx:30:24 main4.go:29: 移除空白符 location.0: {"province":"gd","city":"gz","area":"huangpu"}
2021/06/xx xx:30:24 main4.go:32: 使json 更加容易阅读 pretty location :  {
  "province": "gd",
  "city": "sz",
  "area": "nanshan"
}

2021/06/xx xx:30:24 main4.go:34:  输出整个json this :  {
  "author":{"name":"xiaomotong", "nick": "xiaozhu"},
  "age": 18,
  "hobby": ["play", "eat", "drink"],
  "love.music": "one day",
  "location": [
    {"province": "gd", "city":"gz", "area": "huangpu"},
    {"province": "gd", "city":"sz", "area": "nanshan"},
  ]
}

2021/06/xx xx:30:24 main4.go:39: 扁平化 this :  ["小猪1","小猪2", "小猪3"]

Haha, do you think it’s simple?

We can also customize modifiers, let's take a look

Use the function gjson.AddModifier to add the implementation of our custom modifiers. For details, you can take a look at the following

func main() {
   gjson.AddModifier("myTo", func(json, arg string) string {
       // 将字符串修改为大写
      if arg == "title" {
         return strings.ToTitle(json)
      }

      return json
   })

   const json = `{"children": ["hello", "world", "xiaomotong"]}`
   fmt.Println(gjson.Get(json, "children|@myTo:title"))
}

The results are as follows:

["HELLO", "WORLD", "XIAOMOTONG"]

AddModifier binds custom decoration commands to GJSON syntax. This operation is not thread-safe and should be performed before using all other gjson functions.

// AddModifier binds a custom modifier command to the GJSON syntax.
// This operation is not thread safe and should be executed prior to
// using all other gjson function.
func AddModifier(name string, fn func(json, arg string) string) {
    modifiers[name] = fn
}

`Summarize`

Shared what json and gjson represent
Simple use of gjson
gjson check, get the value
gjson of json line
gjson key path matching rules
Modifiers and custom modifiers of gjson

`Welcome to like, follow, favorite`

Friends, your support and encouragement are my motivation to keep sharing and improve quality

Well, this has to end here, once Casbin GO Permissions Management under the

Technology is open, and our mindset should be more open. Embrace the change, live toward the sun, and work hard to move forward.

I’m , the little magic boy, welcome to like and follow the collection, see you next time~

Application and sharing of gjson in GO