C Program Help

Open scanner.h. Read through the TOKEN_TYPE enum, ensuring that all of your token types (as well as INVALID_TOKENand NO_TOKEN_TYPE) are there.

Read the TOKEN_VALUE union. Decide what member of this union (if any) a token of each TOKEN_TYPE will need to populate. Note that TOKEN_TYPES for which the type conveys all necessary information about the token do not need to store any TOKEN_VALUE.

Note the first line in the main function:

freopen(argv[1], "r", stdin);

By now we should be familiar with freopen; we know this line is overwriting the stdin so input can be more easily gathered from elsewhere. argv[1] is the string value of the first program argument; it should be the name of the file we wish to use as input.

We have a provided input file, input.txt. We need argv[1] to be the path to input.txt in order for the input from input.txt to be used.

CLion allows us to provide program arguments in the Run Configurations; open the Run menu, select Edit Configurations, select the scanner executable and in the Program arguments box enter ../input.txt.

Once you are comfortable with both scanner.h and main.c, you're ready to work on your lexer!

Open scanner.c. There are four TODOs in the file.

The intended functionality for freeToken and printToken should be intuitive, and we will not discuss them further here.

The scanner function should emulate your state machine to tokenize inputs, character by character, to create and return a single token. It will do so by reading 1 new character in each iteration of a while loop. It must use these characters (along with the current state of the finite state machine) to determine what state transitions to take, and to determine when to assign value to the token being constructed.

When a token is composed of multiple characters, you'll generally know it's "done" when you encounter a character that is not part of the token being composed (for instance, if your state machine is in the INTEGER_STATE and you encounter a letter, that letter is clearly not part of the integer, so it must be part of the next token.

When this happens, you'll want to put the unused character back on the input stream so it can be reconsidered as the beginning of the following token. Check out the manual entry for ungetc.

The following is a successful sample run for the provided input.txt:

                                            Process finished with exit code 0

Main.c

#include "scanner.h"

int main(int argc, char **argv) { freopen(argv[1], "r", stdin);

TOKEN *token = NULL; printf(" ");

do { freeToken(&token); token = scanner(); printToken(&token); fflush(stdout); } while (token->type != EOF_TOKEN);

freeToken(&token);

printf(" "); exit(EXIT_SUCCESS); }

Scanner.c

#include "scanner.h"

#include #include #include #include

#define BUFFER_SIZE 32

// // clean up the referenced token (if it isn't NULL). // void freeToken(TOKEN **token) { /* * TODO * free the referenced TOKEN *, and any contained data * which requires freeing. * * Then, set the referenced TOKEN * to NULL */ }

bool updateKeywordOrId(TOKEN *token, char *str) { /* * TODO * Check if the collected token is a keyword by comparing its string * value to the string values of the print and repeat keywords. * * If the token is a keyword, change its type to the corresponding * TOKEN_TYPE and return true. * * If the token is not a keyword, change its type to the token type * for identifiers, allocate space for its string value and copy the * string value into the token. */ }

void printToken (TOKEN **token) { /* * TODO * Print the referenced token in a readable format. * Displayed information should include the TOKEN_TYPE, * and also the token's value if applicable. */ }

TOKEN *scanner() { // buffer to store a token's contained characters while it is being tokenized size_t bufferSize = BUFFER_SIZE; char *stringValue = calloc(sizeof(char), bufferSize); int stringValueIndex = 0;

// allocate space for the new token TOKEN *token = (TOKEN *) malloc(sizeof(TOKEN));

// initialize the token type to invalid token->type = NO_TOKEN_TYPE;

// set state machine to starting state STATE state = START_STATE;

char currentChar = '\0';

while (currentChar != EOF && token->type == NO_TOKEN_TYPE) { currentChar = (char) getchar(); /* * TODO * Given the current state of the state machine * and the next character, update the state machine * (and the string value of the token being built, * if applicable). */

// if the buffer is full, double its size if (stringValueIndex >= bufferSize-1) { bufferSize *= 2; char *temp = calloc(sizeof(char), bufferSize); strcpy(temp, stringValue); free(stringValue); stringValue = temp; } }

free(stringValue); return token; }

Scanner.h

#ifndef __SCANNER_H #define __SCANNER_H

#include #include #include #include

typedef enum { NO_TOKEN_TYPE, INVALID_TOKEN, REPEAT_TOKEN, PRINT_TOKEN, IDENT_TOKEN, INT_TOKEN, FLOAT_TOKEN, ASSIGNMENT_TOKEN, LPAREN_TOKEN, RPAREN_TOKEN, ADD_OP_TOKEN, MULT_OP_TOKEN, SEMICOLON_TOKEN, EOF_TOKEN } TOKEN_TYPE;

typedef union { long integer; double floating_point; char *string; char op; } TOKEN_VALUE;

typedef struct token { TOKEN_TYPE type; TOKEN_VALUE val; } TOKEN;

typedef enum { START_STATE = 0, ID_OR_KEYWORD_STATE, INT_STATE, FLOAT_STATE } STATE;

TOKEN *scanner();

void freeToken(TOKEN **);

void printToken(TOKEN **);

#define BUF_SIZE 128 #define MAX_LINE_LENGTH 256

#endif

input.txt

firstvar = 123;

secondVar = 21.;

var3 = 1.5;

repeat (10) var3 = 2 * (firstvar + secondvar) / (firstvar + 2);

repeat (firstvar + 2 * secondvar) print firstvar;