Class Lexer

Namespace

EzrSquared.Syntax

Assembly

ezrSquared-lib.dll

The ezr² Lexer or Tokenizer. The job of the Lexer is to convert the user input (code) into Token objects to be given as the input to the Parser.

public class Lexer

Inheritance

object

Lexer

Constructors

Lexer(string, string)

Creates a new Lexer object.

public Lexer(string file, string script)

Parameters

file string

The file name/path of the script.

script string

The script to be tokenized.

Fields

_currentChar

The character in the Position of the current lexing iteration in the script.

private char _currentChar

Field Value

char

_file

The file name/path of the script.

private readonly string _file

Field Value

string

_position

The Position of the current lexing iteration in the script.

private Position _position

Field Value

Position

_reachedEnd

The value for checking if the Lexer has reached the end of the script.

private bool _reachedEnd

Field Value

bool

_script

The script to be tokenized.

private readonly string _script

Field Value

string

Methods

Advance()

Advances the current Position in the script.

private void Advance()

CompileColon()

Creates Colon and assignment type (AssignmentAddition, AssignmentMultiplication, etc) Token objects.

private Token CompileColon()

Returns

Token

The created Token.

CompileIdentifier(out EzrSyntaxError?)

Creates Identifier, keyword type (KeywordItem, KeywordFunction, etc) and qeyword type (QeywordC, QeywordFd, etc) Token objects.

private Token CompileIdentifier(out EzrSyntaxError? error)

Parameters

error EzrSyntaxError

Any EzrSyntaxError that occurred in the process; null if none occurred.

Returns

Token

The created Token.

CompileNewLines()

Goes through newline characters and creates a single Token object with TokenTypeNewLine.

private Token CompileNewLines()

Returns

Token

The Token object.

CompileNumber()

Goes through digit and period characters and creates a single Token object with TokenTypeFloatingPoint or Integer.

private Token CompileNumber()

Returns

Token

The Token object.

CompileStringLike(out EzrSyntaxError?)

Creates a Token of types String, Character or CharacterList, depending on the enclosing character.

private Token CompileStringLike(out EzrSyntaxError? error)

Parameters

error EzrSyntaxError

Any EzrSyntaxError that occurred in creating the stringlike; null if none occurred.

Returns

Token

The created Token.

ProcessEscapeSequence(StringBuilder, out EzrSyntaxError?)

Processes an escape sequence in a stringlike.

private void ProcessEscapeSequence(StringBuilder builder, out EzrSyntaxError? error)

Parameters

builder StringBuilder

The System.Text.StringBuilder to append the special character to.

error EzrSyntaxError

Any EzrSyntaxError that occurred in the process; null if none occurred.

ProcessUtf16Sequence(out EzrSyntaxError?)

Processes a UTF-16 escaped sequence in a stringlike.

private char[] ProcessUtf16Sequence(out EzrSyntaxError? error)

Parameters

error EzrSyntaxError

Any EzrSyntaxError that occurred in the process; null if none occurred.

Returns

char[]

The UTF-16 character.

ProcessUtf32Sequence(out EzrSyntaxError?)

Processes a UTF-32 escaped sequence in a stringlike.

private string ProcessUtf32Sequence(out EzrSyntaxError? error)

Parameters

error EzrSyntaxError

Any EzrSyntaxError that occurred in the process; null if none occurred.

Returns

string

The UTF-32 character.

ReverseTo(int)

Reverses to the given index.

private void ReverseTo(int index)

Parameters

index int

The index to reverse to.

Remarks

Warning: The Line decrement is hard set to one if the character at index is a newline, and zero otherwise.

SkipComment()

Skips a comment in the ezr² code.

private void SkipComment()

Tokenize(out List<Token>)

Creates a System.Collections.Generic.List<T> of Token objects from the given script.

public EzrSyntaxError? Tokenize(out List<Token> tokens)

Parameters

tokens List<Token>

The created System.Collections.Generic.List<T> of Token objects.

Returns

EzrSyntaxError

Any EzrSyntaxError that occurred in the lexing; null if none occurred.