The Token stream

The token stream is a doubly linked list with special 'EOF' tokens at the beginning and end. The EOF token at the end points to itself as the next token, and the one at the beginning points to itself as the previous token. So, for a correctly formed token stream (which we routinely assume will always be the case) neither previous nor next can ever return null. We can't express that in the type system because those fields will be null temporarily while the list is being built. As a result, the convention throughout the code base, is to suffix those two getters with a bang (!) everywhere they're invoked.

Tokens are typically accessed through the AST.