51工具盒子# linux json 解析工具 jq {#linux-json-解析工具-jq}
本文讲述 Linux 系统的 json 解析工具 jq 的用法。我们要临时解析一段 json 文本，通过需要手动开发程序，实际上没有必要, 因为有更加方便的方法，linux 系统已经提供了 json 解析工具，而且功能很强大。若您的日志是 json 格式写入的，那么利用这款工具，解析日志就会达到事半功倍的效果。
下载地址: jq (opens new window)。
# 安装 jq {#安装-jq}
# 通用安装方法 {#通用安装方法}
提示
国内访问官方提供的下载地址, 下载速度特别慢，不建议使用该方法。
建议查看后续章节, 根据不同的系统, 使用软件仓库安装。
此方法适用于所有 linux 版本, 只要把二进制程序下载到本地, 即可使用。
请前往官方文档 (opens new window), 查看下载地址。
# ubuntu 安装 jq {#ubuntu-安装-jq}
对于 ubuntu 系统, jq 存在于默认仓库中， 安装更为方便，执行如下命令即可。
sudo apt-get install jq
# centos 安装 jq {#centos-安装-jq}
# 1. 安装 EPEL 软件仓库 {#_1-安装-epel-软件仓库}
yum install epel-release
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# yum install epel-release
已加载插件：fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.163.com
解决依赖关系
--> 执行事务检查
---> Package epel-release.noarch 0:6-8 will be 安装
--> 完成依赖关系计算
依赖关系解决
===================================================================================================================================================================
软件包 架构 版本 仓库 大小
===================================================================================================================================================================
正在安装:
epel-release noarch 6-8 extras 14 k
事务概要
===================================================================================================================================================================
Install 1 Package(s)
总下载量：14 k
Installed size: 22 k
确定吗？[y/N]：y
下载软件包：
epel-release-6-8.noarch.rpm | 14 kB 00:00
运行 rpm_check_debug
执行事务测试
事务测试成功
执行事务
Warning: RPMDB altered outside of yum.
正在安装 : epel-release-6-8.noarch 1/1
Verifying : epel-release-6-8.noarch 1/1
已安装:
epel-release.noarch 0:6-8
完毕！
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# 2. 安装 jq {#_2-安装-jq}
yum install jq -y
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# yum install jq -y
已加载插件：fastestmirror
设置安装进程
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.aliyun.com
base | 3.7 kB 00:00
base/primary_db | 4.7 MB 00:00
epel | 4.7 kB 00:00
epel/primary_db | 6.1 MB 00:03
extras | 3.4 kB 00:00
extras/primary_db | 29 kB 00:00
updates | 3.4 kB 00:00
updates/primary_db | 11 MB 00:05
解决依赖关系
--> 执行事务检查
---> Package jq.x86_64 0:1.3-2.el6 will be 安装
--> 完成依赖关系计算
依赖关系解决
===================================================================================================================================================================================
软件包 架构 版本 仓库 大小
===================================================================================================================================================================================
正在安装:
jq x86_64 1.3-2.el6 epel 94 k
事务概要
===================================================================================================================================================================================
Install 1 Package(s)
总下载量：94 k
Installed size: 230 k
下载软件包：
jq-1.3-2.el6.x86_64.rpm | 94 kB 00:00
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
Importing GPG key 0x0608B895:
Userid : EPEL (6) <epel@fedoraproject.org>
Package: epel-release-6-8.noarch (@extras)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
运行 rpm_check_debug
执行事务测试
事务测试成功
执行事务
正在安装 : jq-1.3-2.el6.x86_64 1/1
Verifying : jq-1.3-2.el6.x86_64 1/1
已安装:
jq.x86_64 0:1.3-2.el6
完毕！
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# jq 常用操作 {#jq-常用操作}
# json 格式化 {#json-格式化}
-
jq 可实现 json 文本的格式化, 详细过程如下
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json {"level":"INFO","time":"2020-08-06 20:49:59.132","caller":"controller/beauty_controller.go:109","message":"添加图片文件的disable后缀, 完成","serviceName":"beauty2","filePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg","newFilePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable"} IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json | jq . { "newFilePath": "/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable", "filePath": "/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg", "serviceName": "beauty2", "message": "添加图片文件的disable后缀, 完成", "caller": "controller/beauty_controller.go:109", "time": "2020-08-06 20:49:59.132", "level": "INFO" } IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# 根据 key 解析出 value 值 {#根据-key-解析出-value-值}
语法: jq .key 的名称
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json
{"level":"INFO","time":"2020-08-06 20:49:59.132","caller":"controller/beauty_controller.go:109","message":"添加图片文件的disable后缀, 完成","serviceName":"beauty2","filePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg","newFilePath":"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable"}
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2# head -n 1 json_file.json | jq .newFilePath
"/data/images/images2/568/224_43947c0eca8baf42190dd09b7730c7fe_75.jpg.disable"
IDC::uy01-05-v19 root@uy01-05-v19:/data/log/beauty2#
# 解析出指定的一组json属性 {#解析出指定的一组json属性}
先查看下内容完整的一行日志
[root@bj-s-21-v10 ~]# grep "用户访问日志, 完成" /data/log/newsBack/info.log | head -n 2
{"level":"INFO","time":"2020-12-10 09:35:41.579","caller":"middleware/api_log_middleware.go:23","message":"用户访问日志, 完成","serviceName":"news-back","uri":"/api/distributor/back/findByCondition?titleLike=","referer":"http://172.18.100.177:8004/distributor/distributorConfig","ip":"172.18.100.177","ipLocation":"局域网 对方和您在同一内部网","duration":"8.513198ms"}
{"level":"INFO","time":"2020-12-10 09:35:44.781","caller":"middleware/api_log_middleware.go:23","message":"用户访问日志, 完成","serviceName":"news-back","uri":"/api/distributor/back/getRevisitRedirectConfig?did=1134019964852768768","referer":"","ip":"172.18.100.177","ipLocation":"局域网 对方和您在同一内部网","duration":"2.454011ms"}
[root@bj-s-21-v10 ~]#
使用逗号,表示要查询哪些json属性
[root@bj-s-21-v10 ~]# grep "用户访问日志, 完成" /data/log/newsBack/info.log | head -n 2 | jq .time,.message,.duration
"2020-12-10 09:35:41.579"
"用户访问日志, 完成"
"8.513198ms"
"2020-12-10 09:35:44.781"
"用户访问日志, 完成"
"2.454011ms"
[root@bj-s-21-v10 ~]#
# 解析 json 日志 {#解析-json-日志}
-
批量解析 json 日志文件的内容
对于 json 日志文件，每一行都是一个 json 对象，jq 支持一次性解析多行。(py3.6) wangshibiao@wangshibiao:~$ head -n 3 /tmp/log/gf-app/2020-11-13.log {"context":{"requestId":"1nzbby6og30c71wvh2dmf6q500ubgeuo"},"level":"info","pTitle":"获取estar的视频广告","sTitle":"estar接口有返回下载广告","time":"2020-11-13 14:11:02"} {"context":{"requestId":"1nzbby6og30c71wvh2dmf6q500ubgeuo"},"level":"info","pTitle":"获取estar的视频广告","sTitle":"estar接口有返回广告数据","time":"2020-11-13 14:11:02"} {"context":{"requestId":"1nzbby6og30c71wvh2dmf6q500ubgeuo"},"level":"info","pTitle":"获取estar的视频广告","sTitle":"完成","time":"2020-11-13 14:11:02"} (py3.6) wangshibiao@wangshibiao:~$ head -n 3 /tmp/log/gf-app/2020-11-13.log | jq { "context": { "requestId": "1nzbby6og30c71wvh2dmf6q500ubgeuo" }, "level": "info", "pTitle": "获取estar的视频广告", "sTitle": "estar接口有返回下载广告", "time": "2020-11-13 14:11:02" } { "context": { "requestId": "1nzbby6og30c71wvh2dmf6q500ubgeuo" }, "level": "info", "pTitle": "获取estar的视频广告", "sTitle": "estar接口有返回广告数据", "time": "2020-11-13 14:11:02" } { "context": { "requestId": "1nzbby6og30c71wvh2dmf6q500ubgeuo" }, "level": "info", "pTitle": "获取estar的视频广告", "sTitle": "完成", "time": "2020-11-13 14:11:02" } (py3.6) wangshibiao@wangshibiao:~$
-
对 json 日志文件实时解析
可以实时解析 json 日志。(py3.6) wangshibiao@wangshibiao:~$ tail -f -n 1 /tmp/log/gf-app/2020-11-13.log | jq { "context": { "requestId": "1nzbby6og30c71wxeew73vn800ore07s" }, "level": "info", "pTitle": "跟踪estar事件接口", "para": { "eventType": "xad_impression", "platform": "android", "adId": 170, "imei": "869941043279996", "slotId": "cHJpb3JpdHlfYWRfc2xvdA==", "action": 5 }, "query": "event_type=xad_impression&package_name=com.zhangshangjianzhi.newapp&scop=None&userId=1315171366231&campaign_id=171&osv=0.1&isPlugin=1&git_last_time=20200318201149&box_pkg_name=c.l.a&ad_system=priority_ad&version_name=5.6.0.5&m_name=rtb&yid=1315171366231&cln=adx-14&deep_link=0&m_version=support_qq_public&total_ram=3GB&cpu_core_num=8&make=meizu&lon=-1&dsp=adx&platform=android&sdk_version=None&ad_user_id=105&time_stamp=20200320135422&device_id=6205ea67-ad97-3c34-97df-da5d3a9fc4e8&imsi&fp=-1&ad_id=170&sd_slot_id=103&city=%E5%86%85%E7%BD%91IP&brand=alps&androidid=81b4f0b18d137a02&des_pla=0&mac=02%3A00%3A00%3A00%3A00%3A00&hn=down.doumistatic.com&autp=7&imei=869941043279996&rp=-1&indus=ep&req_from=3&lat=-1&network=1&git_ci_time=3599&cop=None&cam_rt=14&tg_i=-1&gender&supr=0&ecs_name=unknown&git_br_name=support_qq_public&asign=a16f388884c2e1209636aacee144f765&slot_id=cHJpb3JpdHlfYWRfc2xvdA%3D%3D&fst=bhy&host_name=JS10XXX.local&request_id=3f6825f3-6a6f-11ea-b281-04d9f51f3141&action=5&sucop=None", "sTitle": "", "time": "2020-11-13 14:13:28" }
-
解析 json 日志时，仅打印指定的字段
请见如下示例中tail -f /tmp/log/gf-app/2020-11-13.log | jq .para和tail -f /tmp/log/gf-app/2020-11-13.log | jq .para.eventType的打印结果。(py3.6) wangshibiao@wangshibiao:~$ tail -f /tmp/log/gf-app/2020-11-13.log | jq { "context": { "requestId": "1nzbby6og30c71xbvyyn8k3900evnqa7" }, "level": "info", "pTitle": "跟踪estar事件接口", "para": { "eventType": "xad_impression", "platform": "android", "adId": 170, "imei": "869941043279996", "slotId": "cHJpb3JpdHlfYWRfc2xvdA==", "action": 5 }, "query": "event_type=xad_impression&package_name=com.zhangshangjianzhi.newapp&scop=None&userId=1315171366231&campaign_id=171&osv=0.1&isPlugin=1&git_last_time=20200318201149&box_pkg_name=c.l.a&ad_system=priority_ad&version_name=5.6.0.5&m_name=rtb&yid=1315171366231&cln=adx-14&deep_link=0&m_version=support_qq_public&total_ram=3GB&cpu_core_num=8&make=meizu&lon=-1&dsp=adx&platform=android&sdk_version=None&ad_user_id=105&time_stamp=20200320135422&device_id=6205ea67-ad97-3c34-97df-da5d3a9fc4e8&imsi&fp=-1&ad_id=170&sd_slot_id=103&city=%E5%86%85%E7%BD%91IP&brand=alps&androidid=81b4f0b18d137a02&des_pla=0&mac=02%3A00%3A00%3A00%3A00%3A00&hn=down.doumistatic.com&autp=7&imei=869941043279996&rp=-1&indus=ep&req_from=3&lat=-1&network=1&git_ci_time=3599&cop=None&cam_rt=14&tg_i=-1&gender&supr=0&ecs_name=unknown&git_br_name=support_qq_public&asign=a16f388884c2e1209636aacee144f765&slot_id=cHJpb3JpdHlfYWRfc2xvdA%3D%3D&fst=bhy&host_name=JS10XXX.local&request_id=3f6825f3-6a6f-11ea-b281-04d9f51f3141&action=5&sucop=None", "sTitle": "", "time": "2020-11-13 14:32:23" }
^C (py3.6) wangshibiao@wangshibiao:~$ tail -f /tmp/log/gf-app/2020-11-13.log | jq .para { "eventType": "xad_impression", "platform": "android", "adId": 170, "imei": "869941043279996", "slotId": "cHJpb3JpdHlfYWRfc2xvdA==", "action": 5 }
^C (py3.6) wangshibiao@wangshibiao:~$ tail -f /tmp/log/gf-app/2020-11-13.log | jq .para.eventType "xad_impression"
-
从日志中过滤指定字段，且每条日志一行显示
示例如下：./jq-linux64 -c '{ts,level,msg}'
-c表示每条日志一行显示
{ts,level,msg}用来过滤指定的字段
-
从日志中排除指定字段
有些时候，每条日志中可能会包含大量的信息，又臭又长，这时我们可能希望排除不需要看到的日志内容。使用del函数即可解决，示例如下:./jq-linux64 -c 'del(.level,."User-Agent",.caller)'
> 若某个字段包含-, 那么需要用双引号引起来(见如上示例)，否则报语法错误。
![[最新教程] 设置Telegram(电报) 隐私与安全选项](https://img1.51tbox.com/static/2024-11-23/ofFCdhCVQzUx.jpg)
