We want to build a tokenizer for simple expressions such as xpr = "res = 3 + x_sum*11". Such expressions comprise only three tokens, as follows: (1) Integer literals: one or more digits e.g., 3 11; (2) Identifiers: strings starting with a letter or an underscore and followed by more letters, digits, or underscores e.g., res x_sum; (3) Operators: = + * . Leading or trailing whitespace characters should be skipped.

(e) We can improve the approach further, using another feature which is the scanner() method of regular expressions: It creates a scanner object and attaches it to a string, keeps track of the current position, and moves forward after each successful match. Rewrite the tokenize generator to make use of this feature (using scanner() and calling match()repeatedly, yielding lexeme and token pairs) and again produce the same output as before.