EzrSquared.Syntax.Lexer Class Reference

The ezr² Lexer or Tokenizer. The job of the Lexer is to convert the user input (code) into Token objects to be given as the input to the Parser. More...

Public Member Functions
	Lexer (string file, string script)
	Creates a new Lexer object.
SyntaxError?	Tokenize (out List< Token > tokens)
	Creates a List<T> of Token objects from the given script.

Private Member Functions
void	Advance ()
	Advances the current Position in the script.
void	ReverseTo (int index)
	Reverses to the given index.
void	SkipComment ()
Token	CompileNumber ()
	Goes through digit and period characters and creates a single Token object with TokenType TokenType.FloatingPoint or TokenType.Integer.
Token	CompileNewLines ()
	Goes through newline characters and creates a single Token object with TokenType TokenType.NewLine.
Token	CompileStringLike (out SyntaxError? error)
	Creates a Token of types TokenType.String, TokenType.Character or TokenType.CharacterList, depending on the enclosing character.
void	ProcessEscapeSequence (StringBuilder builder, ref SyntaxError? error)
	Processes an escape sequence in a stringlike.
char[]	ProcessUtf16Sequence (ref SyntaxError? error)
	Processes a UTF-16 escaped sequence in a stringlike.
string	ProcessUtf32Sequence (ref SyntaxError? error)
	Processes a UTF-32 escaped sequence in a stringlike.
Token	CompileColon ()
	Creates TokenType.Colon and assignment type (TokenType.AssignmentAddition, TokenType.AssignmentMultiplication, etc) Token objects.
Token	CompileIdentifier ()
	Creates TokenType.Identifier, keyword type (TokenType.KeywordItem, TokenType.KeywordFunction, etc) and qeyword type (TokenType.QeywordC, TokenType.QeywordFd, etc) Token objects.

Private Attributes
readonly string	_file
	The file name/path of the script.
readonly string	_script
	The script to be tokenized.
readonly Position	_position
	The Position of the current lexing iteration in the script.
char	_currentChar
	The character in the Position of the current lexing iteration in the script.
bool	_reachedEnd
	The value for checking if the Lexer has reached the end of the script.

Detailed Description

The ezr² Lexer or Tokenizer. The job of the Lexer is to convert the user input (code) into Token objects to be given as the input to the Parser.

Constructor & Destructor Documentation

Lexer()

EzrSquared.Syntax.Lexer.Lexer	(	string	file,
		string	script )

Creates a new Lexer object.

Parameters

file	The file name/path of the script.
script	The script to be tokenized.

Member Function Documentation

CompileColon()

Token EzrSquared.Syntax.Lexer.CompileColon ( )

private

Creates TokenType.Colon and assignment type (TokenType.AssignmentAddition, TokenType.AssignmentMultiplication, etc) Token objects.

Returns

The created Token.

CompileIdentifier()

Token EzrSquared.Syntax.Lexer.CompileIdentifier ( )

private

Creates TokenType.Identifier, keyword type (TokenType.KeywordItem, TokenType.KeywordFunction, etc) and qeyword type (TokenType.QeywordC, TokenType.QeywordFd, etc) Token objects.

Returns

The created Token.

CompileNewLines()

Token EzrSquared.Syntax.Lexer.CompileNewLines ( )

private

Goes through newline characters and creates a single Token object with TokenType TokenType.NewLine.

Returns

The Token object.

CompileNumber()

Token EzrSquared.Syntax.Lexer.CompileNumber ( )

private

Goes through digit and period characters and creates a single Token object with TokenType TokenType.FloatingPoint or TokenType.Integer.

Returns

The Token object.

CompileStringLike()

Token EzrSquared.Syntax.Lexer.CompileStringLike ( out SyntaxError? error )

private

Creates a Token of types TokenType.String, TokenType.Character or TokenType.CharacterList, depending on the enclosing character.

Parameters

error

Any SyntaxError that occurred in creating the stringlike; null if none occurred.

Returns

The created Token.

ProcessEscapeSequence()

void EzrSquared.Syntax.Lexer.ProcessEscapeSequence ( StringBuilder builder, ref SyntaxError? error )

private

Processes an escape sequence in a stringlike.

Parameters

builder	The StringBuilder to append the special character to.
error	Any SyntaxError that occurred in the process; null if none occurred.

ProcessUtf16Sequence()

char[] EzrSquared.Syntax.Lexer.ProcessUtf16Sequence ( ref SyntaxError? error )

private

Processes a UTF-16 escaped sequence in a stringlike.

Parameters

error

Any SyntaxError that occurred in the process; null if none occurred.

Returns

The UTF-16 character.

ProcessUtf32Sequence()

string EzrSquared.Syntax.Lexer.ProcessUtf32Sequence ( ref SyntaxError? error )

private

Processes a UTF-32 escaped sequence in a stringlike.

Parameters

error

Any SyntaxError that occurred in the process; null if none occurred.

Returns

The UTF-32 character.

ReverseTo()

void EzrSquared.Syntax.Lexer.ReverseTo ( int index )

private

Reverses to the given index.

Warning: The Position.Line decrement is hard set to one if the character at index is a newline, and zero otherwise.

Parameters

index

The index to reverse to.

Tokenize()

SyntaxError? EzrSquared.Syntax.Lexer.Tokenize

(

out List< Token >

tokens

)

Creates a List<T> of Token objects from the given script.

Parameters

tokens

The created List<T> of Token objects.

Returns

Any SyntaxError that occurred in the lexing; null if none occurred.

---

The documentation for this class was generated from the following file:

• D:/Code/csharp/ezrSquared/src/Syntax/Lexer.cs