A simple tokenizer

Introduction

You have joined a new team as a C++ developer. Your first task is to make a simple but efficient string tokenization engine. There is no place for dynamic memory allocations or heavy object copies here.

Problem Statement

The Input format

1.The input for a tokenizer is either empty or contains a sequence of tokens.

2.Each token consists of akey=valuepair, wherekeyis always an integral value, whereas=is a literal, andvalueis any string.

3.Tokens in the input text are separated by a single-character separator (i.e.,;).

4.The last token can be also followed by the same separator.

5.There are no empty tokens, which means that1=value1;;3=value3is an illegal input.

6.Input sequence will never contain a whitespace next to the=literal or a separator.

The interface

In order to make the tokenizer stateless and not to force it to make a container of tokens, which would demand dynamic memory allocation, the interface requires providing anOutputFunc.

template<char Separator, typename OutputFunc>

void tokenize(std::string_view txt, OutputFunc out);

OutputFuncis a functor providing the following interface:

void(int key, std::string_view value)

This functor has to be called for every token parsed from an input stream.

Use case example

The above interface allows the following use case:

tokenize<';'>("1=value1;2=value2", [](int key, std::string_view value) {

std::cout << "[key = " << key << ", value = " << value << "]";

});

Hints

1.Implement your tokenizer intokenizer.handtokenizer.cppfiles.

2.It might be a good idea to add your own custom unit tests to theunit_tests.cppfile.