in python,

note: if the tokenizer were to run into a character $"",$ treat it as a non$-$alphanumeric character. the only valid characters are a$-$z$,$ A$-$z$,$ and $0-9$

Do NOT use regular expressions or import a tokenizer