Jq

jq is a Command line JSON processor

install

• GNU/Linux: apt install jq
• Mac: brew install jq

Usage

Basic:prettyfier

jq is often used to pretty (make readable) JSON files or request in the shell terminal

For instances, the reply following GET Request to the PZI Wiki api: asking for the first 5 user, will look quite difficult to read

request:

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"

response:

{"query-continue":{"allusers":{"aufrom":"Alef8"}},"query":{"allusers":[{"userid":"234","name":")biyibiyibiyi("},{"userid":"283","name":"0885629"},{"userid":"275","name":"0998247"},{"userid":"160","name":"10000BL"},{"userid":"15","name":"Albert Jongstra"}]}}

But with jq with get the JSON indented according to the hierarchy of the response request:

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5" | jq

response:

{
  "query-continue": {
    "allusers": {
      "aufrom": "Alef8"
    }
  },
  "query": {
    "allusers": [
      {
        "userid": "234",
        "name": ")biyibiyibiyi("
      },
      {
        "userid": "283",
        "name": "0885629"
      },
      {
        "userid": "275",
        "name": "0998247"
      },
      {
        "userid": "160",
        "name": "10000BL"
      },
      {
        "userid": "15",
        "name": "Albert Jongstra"
      }
    ]
  }
}

Filters

jq can be used as a command line filter , who's output can be sent to another application by using the '.' instruction

request: Here grep is matching only the lines with "userid"

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.' | grep userid

response:

        
        "userid": "234",
        "userid": "283",
        "userid": "275",
        "userid": "160",
        "userid": "15",

Filters for querying

jq Filters can be used for querying:

Using the following request if would like to receive only the value of the users names.

If we look at the structure of the response we see that each user name is wrapped withing the following structure

"query": {
    "allusers": [
      {
        "userid": "234",
        "name": ")biyibiyibiyi("
      }
  ]
}

Which we could handle in jq by doing the following steps:

• jq '.query' the value of key "query"
• jq '.query.allusers' the value of key "allusers", within the value of key "query", which is a list
• jq '.query.allusers[]' all the items of list "allusers"
  • jq '.query.allusers[0]' only the 1st item of list
  • jq '.query.allusers[0,2]' items 0, 2 of list
  • jq '.query.allusers[1:]' all items but first
  • jq '.query.allusers[:-1]' all items but last
• jq '.query.allusers[].name' the value of key "name" for all items of list "allusers"

request:

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers[].name'

response:

")biyibiyibiyi("
"0885629"
"0998247"
"10000BL"
"Albert Jongstra"

Filter functions

There are a number of functions that can get us info on the type, length, keys of a response, which are proceeded by the | pipe symbol.

What is the type of .query.allusers :

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers | type'

What is the length of .query.allusers list:

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers | length'

What are the keys of .query.allusers[0] list:

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers[0] | keys'

Filter function: select

The selectfilter function narrow down the output to to items that match certain values

From the list os alluser (.query.allusers[0]) I want only the one that has the value "Albert Jongstra" to key "name". In other words select (.name=="Albert Jongstra")

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers[0] | (.name=="Albert Jongstra")'

Filter function: select

Uses regular expressions to match outputs.

And returns an object for each match it finds. Matches have the following fields:

• offset - offset in UTF-8 codepoints from the beginning of the input
• length - length in UTF-8 codepoints of the match
• string - the string that it matched
• captures - an array of objects representing capturing groups.

Match name of user .query.allusers[].name that star with 0 - regex: "^0.*"

curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers[].name | match ("^0.*")'

{
  "offset": 0,
  "length": 7,
  "string": "0885629",
  "captures": []
}
{
  "offset": 0,
  "length": 7,
  "string": "0998247",
  "captures": []
}
</code>

If we want to get only the strings we can simply add <code>.string</code> to the match pattern: <code>("^0.*").string</code> 

<source lang="bash">curl -s "http://pzwiki.wdka.nl/mw-mediadesign/api.php?format=json&action=query&list=allusers&aulimit=5"  | jq '.query.allusers[].name | match ("^0.*").string'

References

• man jq
• https://shapeshed.com/jq-json/
• https://blog.appoptics.com/jq-json/
• https://thoughtbot.com/blog/jq-is-sed-for-json
• https://blog.appoptics.com/jq-json/
• https://zerokspot.com/weblog/2013/07/18/processing-json-with-jq/