Question: Please write code for this program in C++ programming language. This parser will behave as follows: 1) Whitespace (spaces, tabs, newlines) used only to separate

Please write code for this program in C++ programming language.

This parser will behave as follows:

1) Whitespace (spaces, tabs, newlines) used only to separate tokens - otherwise ignored.

2) Tokens will be made up of strings of text separated by punctuation and symbols

3) Single character tokens (punctuation and symbols) may be defined.

Default behavior: treat symbols like any text character. All characters not separated by whitespace are part of the same token. Parser will provide functions (below) which allow the user to define additional single-character tokens.

Tokenizer() // this is the default class constructor - creates a parser with no initial single-character tokens (separate by whitespace " , \t, ").

Tokenizer(const vector& tokens) // class constructor - creates a parser with single-character tokens defined (defines single character tokens).

void setCharacterTokens(const vector& tokens) // resets the parser's single-character tokens to the list of tokens (separates tokens of strings into pieces when you find these characters: punctuation " !, ., \", \', , ", blocks "[, ], {, }, (, )", symbols "#, $, -")

void addCharacterTokens(const vector& newTokens) // adds the provided single-character tokens to the set of tokens this parser recognizes (separates tokens of strings into pieces when you find these characters: punctuation " !, ., \", \', , ", blocks "[, ], {, }, (, )", symbols "#, $, -")

void addCharacterToken(char newToken) // adds the provided single-character token to the set of tokens this parser recognizes (separates tokens of strings into pieces when you find these characters: punctuation " !, ., \", \', , ", blocks "[, ], {, }, (, )", symbols "#, $, -")

vector parse(string input) const // parses the string and returns a vector of tokens in string form (splits the string into pieces. piece and snippet are passed to tokenize)

vector tokenize(string snippet) // breaks a whitespace-free snippet into tokens by finding and splitting at single-character tokens (checks existence of special characters to further cut)

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!