The set of legal tokens of Core are: Reserved words (11 reserved words): program, begin, end,...

 

Transcribed Image Text:

The set of legal tokens of Core are: Reserved words (11 reserved words): program, begin, end, int, if, then, else, while, loop, read, write Special symbols (19 special symbols): ; , = ! [ ] && || ( ) + - * != == < > <= >= Integers (unsigned) Identifiers: start with uppercase letter, followed by zero or more uppercase letters and zero or more digits. Note that something like "ABCend" is illegal as an id because of the lowercase letters; and it is not two tokens because of lack of whitespace. But ABC123 and A1B2C3 are both legal. White space requirement: White space, i.e., one or more blanks or tab characters or carriage returns, is required between any pair of tokens unless one or both of them is/are special symbols. If one or both of them is/are special symbols. white space is optional. You should not treat white space as a regular token. We will number these tokens 1 through 11 for the reserved words, 12 through 30 for the special symbols, 31 for integer, and 32 for identifier. Note that 31 just tells us that the token in question is an integer, not the actual value of the integer. Similarly, 32 just tells us that the particular token is an identifier, not the name of the identifier. We will also use two additional numbers, 33 and 34, to indicate not actual tokens but situations that the Scanner has to deal with. We will use 33 to indicate that the Scanner is at the end-of-file so there are no more tokens. And 34 will indicate that the Scanner came across a string of characters in the input stream that is not a legal token, in other words, an error token. The Scanner's interface should consist of the following (public) methods: Constructor (): The Scanner's constructor should receive, as parameter, the name of the input file that contains the string of characters to be tokenized. The constructor should start by appropriately instantiating a buffered reader (if you are using Java) corresponding to that file in the usual manner. The constructor should then call a private method tokenizeLine(). This method will read a line from the input file and convert the sequence of characters in that line into a sequence of Core tokens and save them in a private data structure, possibly an array. It should then set a cursor index that points to the first token in that structure. It would be useful to include, in the same structure, the string of characters making up each token since you will need that for the case of integer tokens and identifier tokens. If a line that tokenizeLine() reads consists entirely of white space characters, it should skip that line and read the next line skipping as many lines as necessary to get to a non-empty line. If the line contains a string of characters that is not a valid token, tokenize Line() should still tokenize up to that point with the next number in its private data structure being 34 to indicate illegal token. tokenizeLine() also has to take account of greedy tokenizing. int getToken(): This returns a number between 1 and 32 if the current token is a proper Core token; 33 if the Scanner is at the end of the file and there are no more tokens; 34 if the Scanner comes across a string that is not a legal token. Note that getToken() does not move the "cursor" forward. So repeated calls to getToken() will return the same value. To move past the current token, the Parser has to call skipToken(). void skipToken(): This method moves the "token cursor" to the next token (but does not return anything). If there are no more tokens in the current line, skipToken() calls tokenizeLine() which will read the next line from the input file and convert the sequence of characters in that line into a sequence of Core tokens and save them in the Scanner's data structure and set the cursor index to point to the first token and return. ● If, when skipToken() is called, the current token is 33, i.e., we are already at the end-of-file, the cursor is not moved since there are no more lines or tokens to read. Similarly, if the current token is 34, i.e., we have an illegal token, the cursor is not moved since we do not go past an illlegal token. Your main() function should receive the name of the input file as a command line argument. It should then call the constructor of Scanner, passing that name as an argument.main() should then repeatedly call getToken(), print the returned token number, and call skipToken() until, after some number of iterations, getToken() returns 33 to indicate end-of-file or 34 to indicate invalid token. In either case, it should print an appropriate message and terminate. main() should print each token number on a separate line. Suppose the input file contains the following Core program: program int X; begin X=25; write X; end Your main() function should produce the following sequence of numbers: 14 32 12 2 32...33 corresponding to the tokens, "program", "int", "X", ";", "begin", "X", ..., EOF, but with each number.| The set of legal tokens of Core are: Reserved words (11 reserved words): program, begin, end, int, if, then, else, while, loop, read, write Special symbols (19 special symbols): ; , = ! [ ] && || ( ) + - * != == < > <= >= Integers (unsigned) Identifiers: start with uppercase letter, followed by zero or more uppercase letters and zero or more digits. Note that something like "ABCend" is illegal as an id because of the lowercase letters; and it is not two tokens because of lack of whitespace. But ABC123 and A1B2C3 are both legal. White space requirement: White space, i.e., one or more blanks or tab characters or carriage returns, is required between any pair of tokens unless one or both of them is/are special symbols. If one or both of them is/are special symbols. white space is optional. You should not treat white space as a regular token. We will number these tokens 1 through 11 for the reserved words, 12 through 30 for the special symbols, 31 for integer, and 32 for identifier. Note that 31 just tells us that the token in question is an integer, not the actual value of the integer. Similarly, 32 just tells us that the particular token is an identifier, not the name of the identifier. We will also use two additional numbers, 33 and 34, to indicate not actual tokens but situations that the Scanner has to deal with. We will use 33 to indicate that the Scanner is at the end-of-file so there are no more tokens. And 34 will indicate that the Scanner came across a string of characters in the input stream that is not a legal token, in other words, an error token. The Scanner's interface should consist of the following (public) methods: Constructor (): The Scanner's constructor should receive, as parameter, the name of the input file that contains the string of characters to be tokenized. The constructor should start by appropriately instantiating a buffered reader (if you are using Java) corresponding to that file in the usual manner. The constructor should then call a private method tokenizeLine(). This method will read a line from the input file and convert the sequence of characters in that line into a sequence of Core tokens and save them in a private data structure, possibly an array. It should then set a cursor index that points to the first token in that structure. It would be useful to include, in the same structure, the string of characters making up each token since you will need that for the case of integer tokens and identifier tokens. If a line that tokenizeLine() reads consists entirely of white space characters, it should skip that line and read the next line skipping as many lines as necessary to get to a non-empty line. If the line contains a string of characters that is not a valid token, tokenize Line() should still tokenize up to that point with the next number in its private data structure being 34 to indicate illegal token. tokenizeLine() also has to take account of greedy tokenizing. int getToken(): This returns a number between 1 and 32 if the current token is a proper Core token; 33 if the Scanner is at the end of the file and there are no more tokens; 34 if the Scanner comes across a string that is not a legal token. Note that getToken() does not move the "cursor" forward. So repeated calls to getToken() will return the same value. To move past the current token, the Parser has to call skipToken(). void skipToken(): This method moves the "token cursor" to the next token (but does not return anything). If there are no more tokens in the current line, skipToken() calls tokenizeLine() which will read the next line from the input file and convert the sequence of characters in that line into a sequence of Core tokens and save them in the Scanner's data structure and set the cursor index to point to the first token and return. ● If, when skipToken() is called, the current token is 33, i.e., we are already at the end-of-file, the cursor is not moved since there are no more lines or tokens to read. Similarly, if the current token is 34, i.e., we have an illegal token, the cursor is not moved since we do not go past an illlegal token. Your main() function should receive the name of the input file as a command line argument. It should then call the constructor of Scanner, passing that name as an argument.main() should then repeatedly call getToken(), print the returned token number, and call skipToken() until, after some number of iterations, getToken() returns 33 to indicate end-of-file or 34 to indicate invalid token. In either case, it should print an appropriate message and terminate. main() should print each token number on a separate line. Suppose the input file contains the following Core program: program int X; begin X=25; write X; end Your main() function should produce the following sequence of numbers: 14 32 12 2 32...33 corresponding to the tokens, "program", "int", "X", ";", "begin", "X", ..., EOF, but with each number.|

Answer rating: 100% (QA)

The legal tokens of Core can be classified into the following categories Reserved wordsThese are words that have a special meaning to the Core compile... View the full answer