When we produce tokens, we produce the same arrays that Ripper does. However, we add a couple of convenience methods onto them to make them a little easier to work with. We delegate all other methods to the array.

Instance Methods

The type of the token.

The location of the token in the source.

The state of the lexer when this token was produced.

The slice of the source that this token represents.