【Linux】jq 命令介绍和使用

介绍

jq 命令主要是和json文件的操作有关的辅助命令，可以实现对于Json数据的切片，过滤，映射和转换，在功能上毫不逊色于awk、sed、grep等命令。

注意：这里的jq和我们认知的前端的那个"jq"是两码事，不要混入前端框架的概念。

jq是用C编写，没有运行时依赖，所以几乎可以运行在任何系统上。可以跨各种常用的操作系统，虽然官方没直接提供CenterOs系统的下载方式，但是可以通过添加epel源的方式处理。

作用

可以让linux命令和shell脚本在处理json数据时变得得心应手。

官方案例演示

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5'

结果如下：

格式化输出：

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'

获取第一个元素：

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'

[zxd@localhost ~]$ curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19313  100 19313    0     0   3131      0  0:00:06  0:00:06 --:--:--  4147
{
  "sha": "cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "node_id": "C_kwDOAE3WVdoAKGNmZjUzMzZlYzcxYjZmZWUzOTZhOTViYjBlNGJlYTM2NWUwY2QxZTg",
  "commit": {
    "author": {
      "name": "Mattias Wadman",
      "email": "[e-mail]",
      "date": "2021-06-09T14:02:22Z"
    },
    "committer": {
      "name": "Nico Williams",
      "email": "[e-mail]",
      "date": "2022-05-26T21:04:32Z"
    },
    "message": "docs: Document repeat(exp)",
    "tree": {
      "sha": "d67d5542df1f16d1a48e1fb75749f60482cd874b",
      "url": "https://api.github.com/repos/stedolan/jq/git/trees/d67d5542df1f16d1a48e1fb75749f60482cd874b"
    },
    "url": "https://api.github.com/repos/stedolan/jq/git/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
    "comment_count": 0,
    "verification": {
      "verified": false,
      "reason": "unsigned",
      "signature": null,
      "payload": null
    }
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "html_url": "https://github.com/stedolan/jq/commit/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/comments",
  "author": {
    "login": "wader",
    "id": 185566,
    "node_id": "MDQ6VXNlcjE4NTU2Ng==",
    "avatar_url": "https://avatars.githubusercontent.com/u/185566?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/wader",
    "html_url": "https://github.com/wader",
    "followers_url": "https://api.github.com/users/wader/followers",
    "following_url": "https://api.github.com/users/wader/following{/other_user}",
    "gists_url": "https://api.github.com/users/wader/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/wader/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/wader/subscriptions",
    "organizations_url": "https://api.github.com/users/wader/orgs",
    "repos_url": "https://api.github.com/users/wader/repos",
    "events_url": "https://api.github.com/users/wader/events{/privacy}",
    "received_events_url": "https://api.github.com/users/wader/received_events",
    "type": "User",
    "site_admin": false
  },
  "committer": {
    "login": "nicowilliams",
    "id": 604851,
    "node_id": "MDQ6VXNlcjYwNDg1MQ==",
    "avatar_url": "https://avatars.githubusercontent.com/u/604851?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/nicowilliams",
    "html_url": "https://github.com/nicowilliams",
    "followers_url": "https://api.github.com/users/nicowilliams/followers",
    "following_url": "https://api.github.com/users/nicowilliams/following{/other_user}",
    "gists_url": "https://api.github.com/users/nicowilliams/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/nicowilliams/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/nicowilliams/subscriptions",
    "organizations_url": "https://api.github.com/users/nicowilliams/orgs",
    "repos_url": "https://api.github.com/users/nicowilliams/repos",
    "events_url": "https://api.github.com/users/nicowilliams/events{/privacy}",
    "received_events_url": "https://api.github.com/users/nicowilliams/received_events",
    "type": "User",
    "site_admin": false
  },
  "parents": [
    {
      "sha": "f2ad9517c72f6267ae317639ab56bbfd4a8653d4",
      "url": "https://api.github.com/repos/stedolan/jq/commits/f2ad9517c72f6267ae317639ab56bbfd4a8653d4",
      "html_url": "https://github.com/stedolan/jq/commit/f2ad9517c72f6267ae317639ab56bbfd4a8653d4"
    }
  ]
}

下载地址：

github地址：

• https://github.com/stedolan/jq
• https://stedolan.github.io/jq/download/

下面介绍比较常用的Ubuntu和CenterOs的安装教程。

Ubuntu 安装

安装方式非常简单，直接按照官方教程即可。

jq 1.5 is in the official Debian and Ubuntu repositories. Install using sudo apt-get install jq.

sudo apt-get install jq

ubuntu@VM-8-8-ubuntu:~$ sudo apt-get install jq
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
..
等待安装完成即可

我们直接输入jq命令验证是否安装成功：

ubuntu@VM-8-8-ubuntu:~$ jq 
jq - commandline JSON processor [version 1.6]

Usage:    jq [options] <jq filter> [file...]
    jq [options] --args <jq filter> [strings...]
    jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.

The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).

For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq

Example:

    $ echo '{"foo": 0}' | jq .
    {
        "foo": 0
    }

For a listing of options, use jq --help.
ubuntu@VM-8-8-ubuntu:~$ yum install epel-release
Command 'yum' not found, did you mean:
  command 'gum' from snap gum (0.12.0)
  command 'yum4' from deb nextgen-yum4 (4.5.2-6)
  command 'num' from deb quickcal (2.4-1)
  command 'zum' from deb perforate (1.2-5.1)
  command 'sum' from deb coreutils (8.32-4.1ubuntu1)
  command 'uum' from deb freewnn-jserver (1.1.1~a021+cvs20130302-7build1)
See 'snap info <snapname>' for additional versions.

CenterOs 安装

使用Centos7.9 版本进行安装稍微麻烦一丢丢，查阅官方介绍发现并没有CenterOs系统的安装方式，这是因为需要添加相关的源才行：

1. 安装EPEL源：

yum install -y epel-release 

2. 安装完EPEL源后，可以查看下jq包是否存在：

yum list jq

[zxd@localhost ~]$ yum list jq
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager
Determining fastest mirrors
epel/x86_64/metalink                                                                            | 7.6 kB  00:00:00     
 * base: mirrors.bupt.edu.cn
 * epel: mirrors.bfsu.edu.cn
 * extras: mirrors.bupt.edu.cn
 * updates: mirrors.bupt.edu.cn
epel                                                                                            | 4.7 kB  00:00:00     
(1/3): epel/x86_64/group_gz                                                                     |  99 kB  00:00:00     
(2/3): epel/x86_64/updateinfo                                                                   | 1.0 MB  00:00:01     
(3/3): epel/x86_64/primary_db                                                                   | 7.0 MB  00:00:03     
rabbitmq_erlang                                                                                                  47/47
rabbitmq_rabbitmq-server                                                                                         98/98
Available Packages
jq.x86_64                                                1.6-2.el7                                                 epe

3. 安装jq：

yum install jq

安装完成之后同样可以直接输入命令jq验证一下。

快速上手

我们可以使用tldr命令查看一些案例用法，tldr是什么可以看这篇文章：[[【Linux】Linux命令快速学习神器tldr、cheat介绍和使用]]

[zxd@localhost ~]$ tldr jq

  jq

  A command-line JSON processor that uses a domain-specific language.
  More information: https://stedolan.github.io/jq/manual/.

  - Execute a specific expression (print a colored and formatted json):
  - 最简单用法，查找JSON 文件并且格式化
    cat path/to/file.json | jq '.'

  - Execute a specific script:
    cat path/to/file.json | jq --from-file path/to/script.jq

  - Pass specific arguments:
    cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" ... '. + $ARGS.named'

  - Print specific keys:
    cat path/to/file.json | jq '.key1, .key2, ...'

  - Print specific array items:
    cat path/to/file.json | jq '.[index1], .[index2], ...'

  - Print all array items/object keys:
    cat path/to/file.json | jq '.[]'

  - Add/remove specific keys:
    cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2", ...}'

简单使用

"." 最简单用法

表达式"."作用是格式化输出JSON：

cat test.json | jq "."

[zxd@localhost ~]$ cat test.json | jq "."
[
  {
    "name": "xxxxxx",
    "url": "http://tool.xxxxx.com",
    "address": {
      "city": "厦门",
      "country": "中国"
    },
    "arrayBrowser": [
      {
        "name": "Google",
        "url": "http://www.google.com"
      },
      {
        "name": "Baidu",
        "url": "http://www.baidu.com"
      }
    ]
  },
  {
    "name": "xxxxxx",
    "url": "http://tool.xxxxx.com",
    "address": {
      "city": "大连",
      "country": "中国"
    },
    "arrayBrowser": [
      {
        "name": "360",
        "url": "http://www.so.com"
      },
      {
        "name": "bing",
        "url": "http://www.bing.com"
      }
    ]
  }
]

[index]

输出列表中的第一个元素，可以使用[index]：

cat test.json | jq ".[0]"

[zxd@localhost ~]$ cat test.json | jq ".[0]"
{
  "name": "xxxxxx",
  "url": "http://tool.chinaz.com",
  "address": {
    "city": "厦门",
    "country": "中国"
  },
  "arrayBrowser": [
    {
      "name": "Google",
      "url": "http://www.google.com"
    },
    {
      "name": "Baidu",
      "url": "http://www.baidu.com"
    }
  ]
}

管道符 |

| 管道符号是非常常用的符号，用于把一个命令的执行结果转叫给另一个命令，就像是流水线加工一样：

这里的示例是访问第一个元素，进而访问嵌套的属性，如.name和.address.city。

cat json.txt | jq '.[0] | {name:.name,city:.address.city}'

[zxd@localhost ~]$ cat test.json | jq '.[0] | {name:.name,city:.address.city}'
{
  "name": "站长工具",
  "city": "厦门"
}

cat test.json | jq '.[0] | {name:.arrayBrowser[1].name,city:.address.city}'

{
  "name": "Baidu",
  "city": "厦门"
}

cat test.json | jq ".[] | {name:.arrayBrowser[1].name,city:.address.city}"

{
  "name": "Baidu",
  "city": "厦门"
}
{
  "name": "bing",
  "city": "大连"
}

输出作为数组

把Jq的输出内容转为数组只需要在对应的位置添加[]一对括号即可：

cat test.json | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"

[zxd@localhost ~]$ cat test.json | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"
[
  {
    "name": "Baidu",
    "city": "厦门"
  },
  {
    "name": "bing",
    "city": "大连"
  }
]

key 映射名称修改

在{}中，冒号前面的名字是映射的名称，你可以任意修改，如下：

cat json.txt | jq "[.[] | {name_001:.arrayBrowser[1].name,city_002:.address.city}]"

[
  {
    "name_001": "Baidu",
    "city_002": "厦门"
  },
  {
    "name_001": "bing",
    "city_002": "大连"
  }
]

参考资料

https://www.jianshu.com/p/6de3cfdbdb0e

扩展阅读

http://www.json.cn/wiki.html

https://stedolan.github.io/jq/tutorial/