I need help with this lab1 creating a tokenizer. All required file for this lab are in the following link http://www1.idc.ac.il/tecs/projects/10/project%2010.zip. It'd be much help if anyone can help me solve this.

First, implement the JackTokenizer module specified in the chapter 10. When applied to a text file containing Jack code, the tokenizer should produce a list of tokens, each printed in a separate line along with its classification: symbol, keyword, identifier, integer constant, or string constant. The classification should be recorded using XML tags. Here is an example:

Source Code	Tokenizer output
if (x < 153) {let city = Paris;}	if ( x < 153 ) { let city = Paris ; }

Note that in the case of string constants, the tokenizer throws away the double quote characters. Thats OK.

The tokenizers output has two features dictated by XML conventions. First, an XML file must be enclosed in some begin and end tags, and thats why the and tags were added to the output. Second, four of the symbols used in the Jack language (<, >, ", &) are also used for XML markup, and thus they cannot appear as data in XML files. To solve the problem, we require the tokenizer to output these tokens as <, >, ", and &, respectively. For example, in order for the text < to be displayed properly in a web browser, the source XML should be written as < .

Testing Your Tokenizer:

Test your tokenizer on the Square Dance and Test Array programs. There is no need to test it on the expressionless version of the former.

For each source file Xxx.jack, have your tokenizer give the output file the name XxxT.xml. Apply your tokenizer to every class file in the test programs, then use the supplied TextComparer utility to compare the generated output to the supplied .xml compare files.

Since the output files generated by your tokenizer will have the same names and extensions as those of the supplied compare files, we suggest putting them in separate directories.