jq: Analyzing JSON data on the command line

Oct 8, 2022 14:45 · 785 words · 4 minute read

jq is an awesome command line tool for working with JSON. You can use it to work with any kind of JSON output, e.g. JSON files or JSON logs. Nowadays, many tools offer JSON exports - you can use jq to extract the data you need, transform it into a different JSON structure, or do data aggegrations on it. It is a perfect tool for explorative data analytics on JSON files.

I’m going to demonstrate some basic jq commands to you. For trying the commands out by yourself, you can either install jq, or you can use the online playground jq play.

As data, I use some sample Caddy HTTP request logs (also called “access logs”). You can find them at the end of this post. Caddy’s logs are in JSON format by default (one JSON object per line), which makes them very easy to process programmatically.1 Here is a jq play link with my demo JSON logs: jqplay.org demo JSON

Some basic JQ Commands

Example for a JQ command in action

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

# NOTE
# I use `tail -f` here, so we can execute this command
# on a log file that receives new lines continuously
# => jq will process these lines as they come.

# COMMANDS
# Per log, output each status code
tail -f log.json | jq ".status"
# Per log, output status, host, and uri as a JSON object
tail -f log.json | jq "{status: .status, host: .request.host, uri: .request.uri }"
# `--compact-output` can be used to print out each JSON object on a single line
tail -f log.json | jq --compact-output "{status: .status, host: .request.host, uri: .request.uri }"
# We can do aggregations. Here, print the number of headers instead of the detailed header list.
tail -f log.json | jq --compact-output "{headers: (.request.headers | length)}"

# NOTE
# `--slurp`: Instead of running the filter for each JSON object
# in the input, read the entire input stream into a large array and
# run the filter just once. 
# So we can e.g. do aggregations on all data together,
# not just per object.

# Get count for level `info`
cat log.json | jq --slurp 'map(select(.level == "info")) | length'

# Get the first error log
cat log.json | jq --slurp 'map(select(.level == "error")) | .[0]'
# Get the last error log
cat log.json | jq --slurp 'map(select(.level == "error")) | .[-1]'
# Get uri & ts of the first error log 
cat log.json | jq --slurp 'map(select(.level == "error")) | .[0] | {uri: .request.uri, ts: .ts}'
# Group by level
cat log.json | jq --slurp "group_by(.level) | map({level: .[0].level, count: length})"

# Note, instead of using `cat`, we could have also written:
jq --slurp '<FILTER>' log.json
# But for me, keeping the `|` syntax makes it easier to switch between `cat` and `tail`.

I hope you now have some ideas on how to analyze JSON data with jq! 🔎

Appendix

This is the Caddy HTTP request log file I’m working with (all requests are from me):

1
2
3
4
5
6
7
8
9

{"level":"info","ts":1653157177.7140534,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"29248","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.003684495,"size":12,"status":200,"resp_headers":{"X-Powered-By":["Express"],"Access-Control-Allow-Origin":["*"],"Content-Type":["text/html; charset=utf-8"],"Content-Length":["12"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"Server":["Caddy"],"Date":["Sat, 21 May 2022 18:19:37 GMT"]}}
{"level":"info","ts":1653157187.907846,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"29248","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.003363846,"size":12,"status":200,"resp_headers":{"X-Powered-By":["Express"],"Server":["Caddy"],"Access-Control-Allow-Origin":["*"],"Content-Type":["text/html; charset=utf-8"],"Content-Length":["12"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"Date":["Sat, 21 May 2022 18:19:47 GMT"]}}
{"level":"info","ts":1653157356.011477,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"30981","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.00957792,"size":12,"status":200,"resp_headers":{"Server":["Caddy"],"X-Powered-By":["Express"],"Access-Control-Allow-Origin":["*"],"Content-Type":["text/html; charset=utf-8"],"Content-Length":["12"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"Date":["Sat, 21 May 2022 18:22:36 GMT"]}}
{"level":"info","ts":1653158770.9674704,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"18886","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"If-None-Match":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.008439535,"size":0,"status":304,"resp_headers":{"Server":["Caddy"],"X-Powered-By":["Express"],"Access-Control-Allow-Origin":["*"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"Date":["Sat, 21 May 2022 18:46:10 GMT"]}}
{"level":"info","ts":1653158774.4871852,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"18886","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br"],"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"If-None-Match":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.01356893,"size":0,"status":304,"resp_headers":{"Server":["Caddy"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"Date":["Sat, 21 May 2022 18:46:14 GMT"],"X-Powered-By":["Express"],"Access-Control-Allow-Origin":["*"]}}
{"level":"error","ts":1653158776.337923,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"18931","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/blub","headers":{"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.001578465,"size":143,"status":404,"resp_headers":{"Server":["Caddy"],"X-Content-Type-Options":["nosniff"],"Content-Length":["143"],"Access-Control-Allow-Origin":["*"],"Date":["Sat, 21 May 2022 18:46:16 GMT"],"X-Powered-By":["Express"],"Content-Security-Policy":["default-src 'none'"],"Content-Type":["text/html; charset=utf-8"]}}
{"level":"info","ts":1653158781.544863,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"18886","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"If-None-Match":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.001629192,"size":0,"status":304,"resp_headers":{"Access-Control-Allow-Origin":["*"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"Date":["Sat, 21 May 2022 18:46:21 GMT"],"X-Powered-By":["Express"],"Server":["Caddy"]}}
{"level":"error","ts":1653158782.7774563,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"18931","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/hello","headers":{"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.001860419,"size":144,"status":404,"resp_headers":{"Server":["Caddy"],"X-Content-Type-Options":["nosniff"],"Content-Length":["144"],"Date":["Sat, 21 May 2022 18:46:22 GMT"],"Content-Security-Policy":["default-src 'none'"],"X-Powered-By":["Express"],"Access-Control-Allow-Origin":["*"],"Content-Type":["text/html; charset=utf-8"]}}
{"level":"info","ts":1653158785.6520493,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"80.187.120.130","remote_port":"18886","proto":"HTTP/2.0","method":"GET","host":"powerpack-api.21solutions.de","uri":"/","headers":{"Cookie":[],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"],"If-None-Match":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"],"Accept-Language":["en-GB,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"powerpack-api.21solutions.de"}},"user_id":"","duration":0.005934088,"size":0,"status":304,"resp_headers":{"Date":["Sat, 21 May 2022 18:46:25 GMT"],"Server":["Caddy"],"X-Powered-By":["Express"],"Access-Control-Allow-Origin":["*"],"Etag":["W/\"c-Lve95gjOVATpfV8EL5X4nxwjKHE\""]}}

You can directly play with these logs on this jqplay.org sheet.

  1. For continuously analyzing web server logs & not writing everything by hand, I recommend GoAccess, a visual web log analyzer, that also supports Caddy’s log file format. ↩︎