Efficient endpoint testing with curl, jq and pup

Sometimes I need to work on web pages whose HTML is dynamically loaded from a JSON endpoint. There’s no better way to verify the module’s behaviour than manual testing, but much of the time, I just need to ensure the correct HTML is rendered. It’s slow to click around the site, so I use curl, but since the HTML is inside JSON, it isn’t feasible to interpret its output.

Fortunately, it’s easy to break this HTML out of its JSON cage. I use two excellent tools, jq and pup, for this kind of work.

Say the JSON endpoint’s output is structured as follows:

{ "data": {  "html" : "<div><h2>whatever</h2><span>hey</span></div>" } }

First, I go to the page I’m working on and open Chrome’s network tab. I manually perform the task that hits the JSON endpoint of interest, I right-click on the relevant call in the network tab, and I select “Copy as curl“, which copies all the relevant cookies and headers necessary to replicate the request.

The curl command will be piped into jq and pup as follows:

curl ((ARGS GO HERE)) | jq -r .data.html | pup -p

The jq call extracts the HTML from the JSON and the -r flag removes the outer quotation marks. The pup call formats the HTML and the -p unescapes the output (otherwise it’ll be full of entities).

The output will look like:


You can use CSS selectors to filter the pup call as follows:

$ cat output.html | pup h2


If you want to take your productivity up another level, you can use this combination with watchman-make as described in this post: watchman-make: focus on your code.

jq and pup are very powerful tools, and even though this example is probably the simplest thing you can do with them it can really accelerate your endpoint testing. You can install both tools through Homebrew. Happy coding!

watchman-make: focus on your code

One of my favourite software development tools is watchman-make. From its homepage:

watchman-make is a convenience tool to help automatically invoke your build tool in response to files changing. It is useful to automate building assets or running tests as you save files during development.

In short: you change your files, and watchman-make runs the command of your choice. I don’t like to switch away from my IDE as I code, so automatically re-running my test commands helps me stay productive and focused. It’s also extremely easy to configure.

Here are some examples of how I use watchman-make.

  • Automatically run JS tests when a JS source file, test, or fixture template changes (aliased to watchjs)
    watchman-make -p \
      --'js/**/*.js' 'test/**/*.js' 'test/**/*.tmpl' \
      --make 'npm run -s' \
      -t 'changed-files-test'
  • Automatically sync CSS changes to my development server (aliased to watchcss)
    watchman-make -p '**/*.css' --make work -t sync

    (work sync is an inhouse sync command at Box)

  • Run a designated phpunit test file when I change a PHP file (aliased to watchphpunit)
    watchman-make -p '**/*.php' --make ./phpunit -t $@

One shortcoming: watchman-make expects the commands it runs to have a target, eg make install or npm test; it errors if you just set a make parameter and no target parameter, eg ./run_all_my_tests. Fortunately you can bypass this by setting the target parameter to '' (the empty string). (If you’re feeling motivated, I filed an issue about this on the project’s GitHub and the maintainers said they’d be amenable to a PR)

Installation instructions for watchman-make are available on the project’s GitHub. Happy coding!