HTML validation with vnu

vnu is an HTML validation checker. It’s extremely easy to use and integrates well with all your other command-line tools. I installed it with brew install vnu – it’s also available through npm and pip.

I like to use it in conjunction with cURL, so I can quickly check the validation status of whatever I’m working on:

curl | vnu -

I prefer the JSON output format, because I can pipe it into jq as follows:

// curl ENDPOINT | vnu --format json - 2>&1 | jq
  "messages": [
      "type": "error",
      "lastLine": 3,
      "firstLine": 1,
      "lastColumn": 70,
      "firstColumn": 1,
      "message": "Start tag seen without seeing a doctype first. Expected “<!DOCTYPE html>”.",
      "extract": "  \n\n        <h2 class=\"popup-heading\" title=\"Blah Blah\"> Embed",
      "hiliteStart": 0,
      "hiliteLength": 74
      "type": "error",
      "lastLine": 3,
      "firstLine": 1,
      "lastColumn": 70,
      "firstColumn": 1,
      "message": "Element “head” is missing a required instance of child element “title”.",
      "extract": "  \n\n        <h2 class=\"popup-heading\" title=\"Blah Blah\"> Embed",
      "hiliteStart": 0,
      "hiliteLength": 74
      "type": "error",
      "lastLine": 3,
      "lastColumn": 1583,
      "firstColumn": 1559,
      "message": "Element “div” not allowed as child of element “label” in this context. (Suppressing further errors from this subtree.)",
      "extract": "n\"><label><div class=\"input-title\"> Width",
      "hiliteStart": 10,
      "hiliteLength": 25

If your HTML is returned from a JSON endpoint, use another jq call to extract it from the JSON:

CURL COMMAND | jq -r .data.html | vnu --format json - 2>&1 | jq .messages

By default, vnu expects full HTML pages, and as such will complain if it’s asked to validate HTML fragments. If you’re validating HTML fragments like in the above examples, you’ll probably see messages like Start tag seen without seeing a doctype first. Expected “<!DOCTYPE html>” or Element “head” is missing a required instance of child element “title”. Don’t worry if this happens – these are only problems if you’re trying to validate a full page.

Combined with your other tools, vnu will help you fix your HTML quickly and efficiently. Enjoy!

IntelliJ Task Management: deal with your tickets like a pro

When I submit a patchset for review, I don’t shut down and wait for the requisite shipits to roll in – I move on to the next item on my backlog. Unfortunately, I’ll probably need to go back to the files in that patchset and act on my peers’ feedback, which means I’ll have to clean up my workspace and reopen the files I had earlier. I’ll want to have my editor split, with the code on one side and the tests on the other, and when I’m done addressing the feedback, I’ll have to clean up again, start the other task’s work again, and cross my fingers that I don’t have revisit that original patchset.

Switching between tasks is taxing, but lucky for us, IntelliJ has Task Management built in. Here’s how you set it up, get started, and make your life easier.

  1. Hit cmd-shift-A and type in “Configure Servers”. There are multiple “Configure Servers” actions, so select the “Tasks & Contexts” one.
    Screen Shot 2016-08-24 at 10.17.02 PM
  2. Hit the + icon and select the issue tracking server where your bugs live. For this example, I’ll use a project on my personal GitHub account, but as shown below, all the major issue tracking systems are supported (I usually use my work’s JIRA instance)
    Screen Shot 2016-08-24 at 10.18.46 PM
  3. Enter your repo credentials and details, test it out, then hit OK.
  4. Now the fun starts. Hit option-shift-t, and the task switcher window will appear. Make sure “Open Task” is selected, and hit Return.
    Screen Shot 2016-08-24 at 10.24.13 PM
  5. Voila! All the open issues from your GitHub project will appear. Select the bug you want to work on and hit Return.
    Screen Shot 2016-08-24 at 10.25.03 PM
  6. You’ll get a few options on how to set up your workspace. I like to start each branch with a clean slate, so I select “Clear current context”, but uncheck it if you’d prefer to keep your current files open. With the “VCS Operations” checkboxes, you can switch to a new or existing branch for your code.
    Screen Shot 2016-08-24 at 10.26.49 PM
  7. Hit “OK” and do your work as normal. If you selected “Clear current context”, all the files that were previously open will be closed.

From now on, whenever you switch tickets, make sure to do “Open Task” from step 4. When you switch back to a ticket you worked on previously, IntelliJ will reopen the files that were open last time the ticket was active. They’ll be right where you left them – it’ll even remember if you had multiple panes open. No more tedious cleanup and setup!

This approach has made context-switching between tasks almost painless for me, and as a result of adopting it I have considerably increased my productivity. If you can relate to the frustration in the first paragraph, I highly recommend you try out the IntelliJ task management functionality. Enjoy!

Efficient endpoint testing with curl, jq and pup

Sometimes I need to work on web pages whose HTML is dynamically loaded from a JSON endpoint. There’s no better way to verify the module’s behaviour than manual testing, but much of the time, I just need to ensure the correct HTML is rendered. It’s slow to click around the site, so I use curl, but since the HTML is inside JSON, it isn’t feasible to interpret its output.

Fortunately, it’s easy to break this HTML out of its JSON cage. I use two excellent tools, jq and pup, for this kind of work.

Say the JSON endpoint’s output is structured as follows:

{ "data": {  "html" : "<div><h2>whatever</h2><span>hey</span></div>" } }

First, I go to the page I’m working on and open Chrome’s network tab. I manually perform the task that hits the JSON endpoint of interest, I right-click on the relevant call in the network tab, and I select “Copy as curl“, which copies all the relevant cookies and headers necessary to replicate the request.

The curl command will be piped into jq and pup as follows:

curl ((ARGS GO HERE)) | jq -r .data.html | pup -p

The jq call extracts the HTML from the JSON and the -r flag removes the outer quotation marks. The pup call formats the HTML and the -p unescapes the output (otherwise it’ll be full of entities).

The output will look like:


You can use CSS selectors to filter the pup call as follows:

$ cat output.html | pup h2


If you want to take your productivity up another level, you can use this combination with watchman-make as described in this post: watchman-make: focus on your code.

jq and pup are very powerful tools, and even though this example is probably the simplest thing you can do with them it can really accelerate your endpoint testing. You can install both tools through Homebrew. Happy coding!

watchman-make: focus on your code

One of my favourite software development tools is watchman-make. From its homepage:

watchman-make is a convenience tool to help automatically invoke your build tool in response to files changing. It is useful to automate building assets or running tests as you save files during development.

In short: you change your files, and watchman-make runs the command of your choice. I don’t like to switch away from my IDE as I code, so automatically re-running my test commands helps me stay productive and focused. It’s also extremely easy to configure.

Here are some examples of how I use watchman-make.

  • Automatically run JS tests when a JS source file, test, or fixture template changes (aliased to watchjs)
    watchman-make -p \
      --'js/**/*.js' 'test/**/*.js' 'test/**/*.tmpl' \
      --make 'npm run -s' \
      -t 'changed-files-test'
  • Automatically sync CSS changes to my development server (aliased to watchcss)
    watchman-make -p '**/*.css' --make work -t sync

    (work sync is an inhouse sync command at Box)

  • Run a designated phpunit test file when I change a PHP file (aliased to watchphpunit)
    watchman-make -p '**/*.php' --make ./phpunit -t $@

One shortcoming: watchman-make expects the commands it runs to have a target, eg make install or npm test; it errors if you just set a make parameter and no target parameter, eg ./run_all_my_tests. Fortunately you can bypass this by setting the target parameter to '' (the empty string). (If you’re feeling motivated, I filed an issue about this on the project’s GitHub and the maintainers said they’d be amenable to a PR)

Installation instructions for watchman-make are available on the project’s GitHub. Happy coding!