Efficient endpoint testing with curl, jq and pup

Sometimes I need to work on web pages whose HTML is dynamically loaded from a JSON endpoint. There’s no better way to verify the module’s behaviour than manual testing, but much of the time, I just need to ensure the correct HTML is rendered. It’s slow to click around the site, so I use curl, but since the HTML is inside JSON, it isn’t feasible to interpret its output.

Fortunately, it’s easy to break this HTML out of its JSON cage. I use two excellent tools, jq and pup, for this kind of work.

Say the JSON endpoint’s output is structured as follows:


{ "data": {  "html" : "<div><h2>whatever</h2><span>hey</span></div>" } }

First, I go to the page I’m working on and open Chrome’s network tab. I manually perform the task that hits the JSON endpoint of interest, I right-click on the relevant call in the network tab, and I select “Copy as curl“, which copies all the relevant cookies and headers necessary to replicate the request.

The curl command will be piped into jq and pup as follows:


curl ((ARGS GO HERE)) | jq -r .data.html | pup -p

The jq call extracts the HTML from the JSON and the -r flag removes the outer quotation marks. The pup call formats the HTML and the -p unescapes the output (otherwise it’ll be full of entities).

The output will look like:


<html>
 <head>
 </head>
 <body>
  <div>
   <h2>
    whatever
   </h2>
   <span>
    hey
   </span>
  </div>
 </body>
</html>

You can use CSS selectors to filter the pup call as follows:


$ cat output.html | pup h2

<h2>
 whatever
</h2>

If you want to take your productivity up another level, you can use this combination with watchman-make as described in this post: watchman-make: focus on your code.

jq and pup are very powerful tools, and even though this example is probably the simplest thing you can do with them it can really accelerate your endpoint testing. You can install both tools through Homebrew. Happy coding!

Leave a Reply