Requirements

You will extend the Lexer to process additional tokens and improve its output.

Requirement $1$

The current implementation of Lexer reads a hardcoded file. Lexer must be updated to take a path to a file as a command line argument:

java lexer.Lexer sample$\_$files$/$simple$.$x

If a filename is not supplied, a usage instruction should be displayed $($in this snippet, the $>$ character indicates the shell prompt and execution command; it is not a part of the required output$)$:

$>$ java lexer.Lexer

usage: java lexer.Lexer filename.x

Requirement $2$

Our compiler must be updated to be able to process additional tokens. The new tokens must be added to the tokens.txt file, and the CompilerTools run to re$-$generate the SymbolTable class and the TokenKind enumeration. Remember from the lecture that we do not want to $($must not$)$ use regular expressions.

Symbolic Constant Lexeme Description of Token

Greater $>$

GreaterEqual $>=$

Number num This is the type keyword used to declare a variable of type num

NumberLit Any literal, which is one or more numbers followed by a decimal point, followed by one or more numbers:

Valid examples: $1\mathrm{.}2123\mathrm{.}546213123123123123\mathrm{.}0$

Invalid examples: $\mathrm{.}21.$

CharType character This is the type keyword used to declare a variable of type character

CharLit Any literal, which is a double quote, followed by any ASCIILinks to an external site. character, followed by a double quote:

Valid examples: $"$a$""\backslash$t$"$

Invalid examples: $"$a $"$bb$"$

From from Keyword

To to Keyword

Step step Keyword

Requirement $3$

The Token class must be updated to include the source file's line number where a token was found.

Requirement $4$

Lexer output must be updated for readability; the line number of the Token and the type of token created must be included. The updated format for a token line is:

$20$ columns, left aligned, for the token's lexeme, followed by a space

"left:" followed by a space

$8$ columns, left aligned, for the left position, followed by a space

"right:" followed by a space

$8$ columns, left aligned, for the right position, followed by a space

"line:" followed by a space

$8$ columns, left aligned, for the line number, followed by a space

The symbol

$($Hint: Use String.format$)$

Requirement $5$

Lexer output must be updated to include a printout of each line, with its line number, read in from the source file. Line numbers should be printed in $3$ columns, right$-$aligned. Note that when an error is encountered, the error should be reported as usual, and the lines of the source file should be output, with line numbers up to and including the error line.

Appendix A

The complete output for the source file simple.x $($the indentation you see for the file output is to allow for three$-$digit line numbers$)$:

$>$ java lexer.Lexer sample$\_$files$/$simple$.$x

program left: $0$ right: $6$ line: $1$ Program

$\{$ left: $8$ right: $8$ line: $1$ LeftBrace

int left: $10$ right: $12$ line: $1$ IntType

i left: $14$ right: $14$ line: $1$ Identifier

int left: $16$ right: $18$ line: $1$ IntType

j left: $20$ right: $20$ line: $1$ Identifier

i left: $3$ right: $3$ line: $2$ Identifier

$=$ left: $5$ right: $5$ line: $2$ Assign

i left: $7$ right: $7$ line: $2$ Identifier

$+$ left: $9$ right: $9$ line: $2$ Plus

j left: $11$ right: $11$ line: $2$ Identifier

$+$ left: $13$ right: $13$ line: $2$ Plus

$7$ left: $15$ right: $15$ line: $2$ IntLit

j left: $3$ right: $3$ line: $3$ Identifier

$=$ left: $5$ right: $5$ line: $3$ Assign

write left: $7$ right: $11$ line: $3$ Identifier

$($ left: $12$ right: $12$ line: $3$ LeftParen

i left: $13$ right: $13$ line: $3$ Identifier

$)$ left: $14$ right: $14$ line: $3$ RightParen

$\}$ left: $0$ right: $0$ line: $4$ RightBrace

$1$: program $\{$ int i int j

$2$: i $=$ i $+$ j $+7$

$3$: j $=$ write$($i$)$

$4$: $\}$

Appendix B

The current output for our lexer looks like this:

$>$ java lexer.Lexer

L: $0,$ R: $6,$ program

L: $8,$ R: $8,\{$

L: $10,$ R: $12,$ int

L: $14,$ R: $14,$ i

L: $16,$ R: $18,$ int

L: $20,$ R: $20,$ j

L: $3,$ R: $3,$ i

L: $5,$ R: $5,=$

L: $7,$ R: $7,$ i

L: $9,$ R: $9,+$

L: $11,$ R: $11,$ j

L: $13,$ R: $13,+$

L: $15,$ R: $15,7$

L: $3,$ R: $3,$ j

L: $5,$ R: $5,=$

L: $7,$ R: $11,$ write

L: $12,$ R: $12,($

L: $13,$ R: $13,$ i

L: $14,$ R: $14,)$

L: $0,$ R: $0,\}>.$github

$>.$vscode

$>$ exceptions

$$ lexer

$$ daos

$J$ Symbol.java

$J$ Token.java

$$ readers

$J$ IReader.java

$$ SourceFileReader.java

$J$ IL