Given an input string, a scanner collects a coherent sequence of characters with the

same character types and saves it as a single unit, called a token. The scanner then saves each

token into a table with two columns: a token string and token type. To simplify the

representation in memory, each token string will contain a maximum of $8$ characters. That way,

each entry in the token table always takes up exactly $12$ bytes $(8$ bytes for a token and a word

for its type$).$

The numeric encoding of token types is identical to the character types used in HW $4,$

but does not include the blank character type any longer since the scanner will not save blank

tokens.

Token type $1--$ Number : $01..9$

Token type $2--$ Variable : An alphabetic letter followed by alpha$-$numeric characters

Token type $3$ Operator : $*+-/$

Token type $4$ Delimiter : $.(),$ : $

Token type $5--$ End of Line: #

For example, given an input string below,