Improve nextToken documentation

[itsjamie]

The parser and tokenizer in nextToken should document the responsibilities between callee/caller for the function so tests can be written to ensure that it parses correctly.

[itsjamie]

I'm interested in doing this because I'd like to implement go-fuzz for this input to ensure behaviour is as expected for each header that is parsed. Thoughts?

[niaow]

Good idea! Send a PR to the dev branch if you get something working.