CompilerKit::Token

Token represents a single source code token, as generated by the lexical analysis phase of your compiler. It keeps track of its location in the source code, the text it represents, and a type (encoded as a string). Some common types are provided as static members of the class:

• Token::Identifier = "Identifier"
• Token::Real = "Real"
• Token::Integer = "Integer"
• Token::EndOfFile = "EndOfFile"
• Token::InvalidToken = "InvalidToken"
• Token::InvalidChar = "InvalidChar"

Public API

Token(const std::string& type, const std::string& text, int line, int col)

Creates a new Token instance for a given text and type, at a given location. While it is possible to create token that way, Scanner provides a more streamlined API for this through Scanner::makeToken().

Parameters:

• type: the type of token to create.
• text: the text the token represents.
• line: the source code line where the token starts.
• column: the source code column where the token starts.

value<Type>() const -> Type

If the this represents a number literal (type() is Token::Real or Token::Integer), returns the numeric value it represents, cast to Type.

Template Parameters:

• Type: the type of numeric value to convert to. Valid types are int and float.

is(const std::string& type) const -> bool

Returns whether this is of the given type.

Parameters:

• type: the type to check this against.

text() const -> const std::string&

Returns the text of this.

type() const -> const std::string&

Returns the type of this.