Java program:

How can I create a scanner (also known as a tokenizer) that will take a sequence of characters (i.e., the source files of the LITTLE language) as input and produce a sequence of tokens? I'm using ANTLR.

For example, this is my input:

And this is what the java tokenizer output should look like:

I'm also using these imports:

import org.antlr.v4.runtime.*; import org.antlr.v4.runtime.tree.*;

PROGRAM fibonacci BEGIN STRING input := "Please input an integer number: "; STRING space :=""; STRING eol := " "; FUNCTION INT F (INT n) BEGIN IF (n>2) RETURN F(n-1)+F(n-2); ELSE RETURN 1; ENDIF END FUNCTION VOID main() BEGIN INT i, end, result; WRITE(input); READ(end); i:= 0; WHILE (i != end) result := F(i); WRITE (i,space); WRITE (result,eol); i:=i+1; ENDWHILE END END Token Type: KEYWORD Value: PROGRAM Token Type: IDENTIFIER Value: fibonacci Token Type: KEYWORD Value: BEGIN Token Type: KEYWORD Value: STRING Token Type: IDENTIFIER Value: input Token Type: OPERATOR Value: := Token Type: STRINGLITERAL Value: "Please input an integer number: Token Type: OPERATOR Value: ; Token Type: KEYWORD Value: STRING Token Type: IDENTIFIER 11 Value: space 11 11 Token Type: OPERATOR Value: := Token Type: STRINGLITERAL Value: Token Type: OPERATOR Value: ; Token Type: KEYWORD Value: STRING Token Type: IDENTIFIER Value: eol Token Type: OPERATOR Value: :=