【Linux】jq 命令介绍和使用

介绍

jq 命令主要是和json文件的操作有关的辅助命令,可以实现对于Json数据的切片,过滤,映射和转换,在功能上毫不逊色于awksedgrep等命令。

注意:这里的jq和我们认知的前端的那个"jq"是两码事,不要混入前端框架的概念。

jq是用C编写,没有运行时依赖,所以几乎可以运行在任何系统上。可以跨各种常用的操作系统,虽然官方没直接提供CenterOs系统的下载方式,但是可以通过添加epel源的方式处理。

作用

可以让linux命令和shell脚本在处理json数据时变得得心应手

官方案例演示

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5'

结果如下:

格式化输出:

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'

获取第一个元素:

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'
[zxd@localhost ~]$ curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19313  100 19313    0     0   3131      0  0:00:06  0:00:06 --:--:--  4147
{
  "sha": "cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "node_id": "C_kwDOAE3WVdoAKGNmZjUzMzZlYzcxYjZmZWUzOTZhOTViYjBlNGJlYTM2NWUwY2QxZTg",
  "commit": {
    "author": {
      "name": "Mattias Wadman",
      "email": "mattias.wadman@gmail.com",
      "date": "2021-06-09T14:02:22Z"
    },
    "committer": {
      "name": "Nico Williams",
      "email": "nico@cryptonector.com",
      "date": "2022-05-26T21:04:32Z"
    },
    "message": "docs: Document repeat(exp)",
    "tree": {
      "sha": "d67d5542df1f16d1a48e1fb75749f60482cd874b",
      "url": "https://api.github.com/repos/stedolan/jq/git/trees/d67d5542df1f16d1a48e1fb75749f60482cd874b"
    },
    "url": "https://api.github.com/repos/stedolan/jq/git/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
    "comment_count": 0,
    "verification": {
      "verified": false,
      "reason": "unsigned",
      "signature": null,
      "payload": null
    }
  },
  "url": "https://api.github.com/repos/stedolan/jq/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "html_url": "https://github.com/stedolan/jq/commit/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8",
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/cff5336ec71b6fee396a95bb0e4bea365e0cd1e8/comments",
  "author": {
    "login": "wader",
    "id": 185566,
    "node_id": "MDQ6VXNlcjE4NTU2Ng==",
    "avatar_url": "https://avatars.githubusercontent.com/u/185566?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/wader",
    "html_url": "https://github.com/wader",
    "followers_url": "https://api.github.com/users/wader/followers",
    "following_url": "https://api.github.com/users/wader/following{/other_user}",
    "gists_url": "https://api.github.com/users/wader/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/wader/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/wader/subscriptions",
    "organizations_url": "https://api.github.com/users/wader/orgs",
    "repos_url": "https://api.github.com/users/wader/repos",
    "events_url": "https://api.github.com/users/wader/events{/privacy}",
    "received_events_url": "https://api.github.com/users/wader/received_events",
    "type": "User",
    "site_admin": false
  },
  "committer": {
    "login": "nicowilliams",
    "id": 604851,
    "node_id": "MDQ6VXNlcjYwNDg1MQ==",
    "avatar_url": "https://avatars.githubusercontent.com/u/604851?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/nicowilliams",
    "html_url": "https://github.com/nicowilliams",
    "followers_url": "https://api.github.com/users/nicowilliams/followers",
    "following_url": "https://api.github.com/users/nicowilliams/following{/other_user}",
    "gists_url": "https://api.github.com/users/nicowilliams/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/nicowilliams/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/nicowilliams/subscriptions",
    "organizations_url": "https://api.github.com/users/nicowilliams/orgs",
    "repos_url": "https://api.github.com/users/nicowilliams/repos",
    "events_url": "https://api.github.com/users/nicowilliams/events{/privacy}",
    "received_events_url": "https://api.github.com/users/nicowilliams/received_events",
    "type": "User",
    "site_admin": false
  },
  "parents": [
    {
      "sha": "f2ad9517c72f6267ae317639ab56bbfd4a8653d4",
      "url": "https://api.github.com/repos/stedolan/jq/commits/f2ad9517c72f6267ae317639ab56bbfd4a8653d4",
      "html_url": "https://github.com/stedolan/jq/commit/f2ad9517c72f6267ae317639ab56bbfd4a8653d4"
    }
  ]
}

下载地址:

github地址:

下面介绍比较常用的Ubuntu和CenterOs的安装教程。

Ubuntu 安装

安装方式非常简单,直接按照官方教程即可。

jq 1.5 is in the official Debian and Ubuntu repositories. Install using sudo apt-get install jq.
sudo apt-get install jq
ubuntu@VM-8-8-ubuntu:~$ sudo apt-get install jq
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
..
等待安装完成即可

我们直接输入jq命令验证是否安装成功:

ubuntu@VM-8-8-ubuntu:~$ jq 
jq - commandline JSON processor [version 1.6]

Usage:    jq [options] <jq filter> [file...]
    jq [options] --args <jq filter> [strings...]
    jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.

The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).

For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq

Example:

    $ echo '{"foo": 0}' | jq .
    {
        "foo": 0
    }

For a listing of options, use jq --help.
ubuntu@VM-8-8-ubuntu:~$ yum install epel-release
Command 'yum' not found, did you mean:
  command 'gum' from snap gum (0.12.0)
  command 'yum4' from deb nextgen-yum4 (4.5.2-6)
  command 'num' from deb quickcal (2.4-1)
  command 'zum' from deb perforate (1.2-5.1)
  command 'sum' from deb coreutils (8.32-4.1ubuntu1)
  command 'uum' from deb freewnn-jserver (1.1.1~a021+cvs20130302-7build1)
See 'snap info <snapname>' for additional versions.

CenterOs 安装

使用Centos7.9 版本进行安装稍微麻烦一丢丢,查阅官方介绍发现并没有CenterOs系统的安装方式,这是因为需要添加相关的源才行:

  1. 安装EPEL源:
yum install -y epel-release 
  1. 安装完EPEL源后,可以查看下jq包是否存在:
yum list jq
[zxd@localhost ~]$ yum list jq
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager
Determining fastest mirrors
epel/x86_64/metalink                                                                            | 7.6 kB  00:00:00     
 * base: mirrors.bupt.edu.cn
 * epel: mirrors.bfsu.edu.cn
 * extras: mirrors.bupt.edu.cn
 * updates: mirrors.bupt.edu.cn
epel                                                                                            | 4.7 kB  00:00:00     
(1/3): epel/x86_64/group_gz                                                                     |  99 kB  00:00:00     
(2/3): epel/x86_64/updateinfo                                                                   | 1.0 MB  00:00:01     
(3/3): epel/x86_64/primary_db                                                                   | 7.0 MB  00:00:03     
rabbitmq_erlang                                                                                                  47/47
rabbitmq_rabbitmq-server                                                                                         98/98
Available Packages
jq.x86_64                                                1.6-2.el7                                                 epe
  1. 安装jq
yum install jq

安装完成之后同样可以直接输入命令jq验证一下。

快速上手

我们可以使用tldr命令查看一些案例用法,tldr是什么可以看这篇文章:[[【Linux】Linux命令快速学习神器tldr、cheat介绍和使用]]

[zxd@localhost ~]$ tldr jq

  jq

  A command-line JSON processor that uses a domain-specific language.
  More information: https://stedolan.github.io/jq/manual/.

  - Execute a specific expression (print a colored and formatted json):
  - 最简单用法,查找JSON 文件并且格式化
    cat path/to/file.json | jq '.'

  - Execute a specific script:
    cat path/to/file.json | jq --from-file path/to/script.jq

  - Pass specific arguments:
    cat path/to/file.json | jq --arg "name1" "value1" --arg "name2" "value2" ... '. + $ARGS.named'

  - Print specific keys:
    cat path/to/file.json | jq '.key1, .key2, ...'

  - Print specific array items:
    cat path/to/file.json | jq '.[index1], .[index2], ...'

  - Print all array items/object keys:
    cat path/to/file.json | jq '.[]'

  - Add/remove specific keys:
    cat path/to/file.json | jq '. +|- {"key1": "value1", "key2": "value2", ...}'

简单使用

"." 最简单用法

表达式"."作用是格式化输出JSON:

cat test.json | jq "."
[zxd@localhost ~]$ cat test.json | jq "."
[
  {
    "name": "xxxxxx",
    "url": "http://tool.xxxxx.com",
    "address": {
      "city": "厦门",
      "country": "中国"
    },
    "arrayBrowser": [
      {
        "name": "Google",
        "url": "http://www.google.com"
      },
      {
        "name": "Baidu",
        "url": "http://www.baidu.com"
      }
    ]
  },
  {
    "name": "xxxxxx",
    "url": "http://tool.xxxxx.com",
    "address": {
      "city": "大连",
      "country": "中国"
    },
    "arrayBrowser": [
      {
        "name": "360",
        "url": "http://www.so.com"
      },
      {
        "name": "bing",
        "url": "http://www.bing.com"
      }
    ]
  }
]

[index]

输出列表中的第一个元素,可以使用[index]:

cat test.json | jq ".[0]"
[zxd@localhost ~]$ cat test.json | jq ".[0]"
{
  "name": "xxxxxx",
  "url": "http://tool.chinaz.com",
  "address": {
    "city": "厦门",
    "country": "中国"
  },
  "arrayBrowser": [
    {
      "name": "Google",
      "url": "http://www.google.com"
    },
    {
      "name": "Baidu",
      "url": "http://www.baidu.com"
    }
  ]
}

管道符 |

| 管道符号是非常常用的符号,用于把一个命令的执行结果转叫给另一个命令,就像是流水线加工一样:

这里的示例是访问第一个元素,进而访问嵌套的属性,如.name.address.city

cat json.txt | jq '.[0] | {name:.name,city:.address.city}'
[zxd@localhost ~]$ cat test.json | jq '.[0] | {name:.name,city:.address.city}'
{
  "name": "站长工具",
  "city": "厦门"
}
cat test.json | jq '.[0] | {name:.arrayBrowser[1].name,city:.address.city}'
{
  "name": "Baidu",
  "city": "厦门"
}
cat test.json | jq ".[] | {name:.arrayBrowser[1].name,city:.address.city}"
{
  "name": "Baidu",
  "city": "厦门"
}
{
  "name": "bing",
  "city": "大连"
}

输出作为数组

把Jq的输出内容转为数组只需要在对应的位置添加[]一对括号即可:

cat test.json | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"
[zxd@localhost ~]$ cat test.json | jq "[.[] | {name:.arrayBrowser[1].name,city:.address.city}]"
[
  {
    "name": "Baidu",
    "city": "厦门"
  },
  {
    "name": "bing",
    "city": "大连"
  }
]

key 映射名称修改

在{}中,冒号前面的名字是映射的名称,你可以任意修改,如下:

cat json.txt | jq "[.[] | {name_001:.arrayBrowser[1].name,city_002:.address.city}]"
[
  {
    "name_001": "Baidu",
    "city_002": "厦门"
  },
  {
    "name_001": "bing",
    "city_002": "大连"
  }
]

参考资料

https://www.jianshu.com/p/6de3cfdbdb0e

扩展阅读

http://www.json.cn/wiki.html

https://stedolan.github.io/jq/tutorial/


技术读书笔记
个人阅读过的技术数据整理

赐他一块白石,石上写着新名

172 声望
34 粉丝
0 条评论
推荐阅读
《跟闪电侠学Netty》阅读笔记 - 开篇入门Netty
和 《Netty In Action》 不同,这本书直接从Netty入门程序代码开始引入Netty框架,前半部分教你如何用Netty搭建简易的通讯系统,整体难度比较低,后半部分直接从服务端源码、客户端源码、ChannelPipeline开始介绍...

Xander阅读 639

Laravel入门及实践,快速上手ThinkSNS+二次开发
【摘要】自从ThinkSNS+不使用ThinkPHP框架而使用Laravel框架之后,很多人都说技术门槛抬高了,其实你与TS+的距离仅仅只是学习一个新框架而已,所以,我们今天来说说Laravel的入门。

ThinkSNS1阅读 2.5k

C++编译器和链接器的完全指南
C++是一种强类型语言,它的编译和链接是程序开发过程中不可或缺的两个环节。编译器和链接器是两个非常重要的概念。本文将详细介绍C++中的编译器和链接器以及它们的工作原理和使用方法。

小万哥2阅读 1.1k

封面图
confluence7.2.1的部署与迁移---呕心沥血版
Confluence是一个专业的企业知识管理与协同软件,也可以用于构建企业wiki。使用简单,但它强大的编辑和站点管理特征能够帮助团队成员之间共享信息、文档协作、集体讨论,信息推送。 到官网下载最新版本,截止目前...

暗涌阅读 7.9k

Ubuntu 20.04 读写 Windows 10 共享目录(qbit)
本文实验环境中,Windows 操作系统版本为 Windows Server 2016,应该对 Windows 10 和 Windows Server 2019 同样适用

qbit1阅读 6.3k

Ubuntu 20.04 搭建 Elasticsearch 7.x 小集群(qbit)
环境ES 节点硬件:3 台 AWS m5.4xlarge(16 vCPU/64GB 内存)Kibana 硬件:1 台 AWS m5.large(2 vCPU/8GB 内存)操作系统:Ubuntu 20.04 LTSElasticsearch 7.9.3Kibana 7.9.3机器示意图操作系统这里主要讲 EBS ...

qbit阅读 4.1k

ubuntu 常见操作笔记
PHP CLIsudo sed -i 's/display_errors = .*/display_errors = On/' /etc/php/7.4/cli/php.inisudo sed -i 's/error_reporting = .*/error_reporting = E_ALL/' /etc/php/7.4/cli/php.inisudo sed -i 's/memory_l...

zhiqiang1阅读 1.5k

赐他一块白石,石上写着新名

172 声望
34 粉丝
宣传栏