Question: Q2. Tokenization function: Write a function that gets a sentence or a text as its input parameter, breaks it into separate words and returns a

Q2. Tokenization function: Write a function that gets a sentence or a text as its input parameter, breaks it into separate words and returns a list of the words extracted from the text. For example, let the function's name be "tokenize", then if the following sentence is passed to the function: "this is my first class, I love my class". The returning result must be: [this, is, my, first, class, , I, love, my, class In other words, if we call the method as show below: tokenize("this is my first class, I love my class") the following result will be returned: ['this', 'is', 'my', 'first', 'class,', 'I', 'love', 'my', 'class'] Every single element of the list is an individual word extracted from the text. All Words in the text are separated based on spaces between them (if there is a space between two consecutive words, they must be separated and listed as two different words). As a result, "data set" will be treated as two separate words rather than a single word. Note: for the sake of simplicity we assume that there is exactly one space between every two consecutive words (words in the sentence are separated by exactly one space, because if there is more than one space between each pair of words then It makes the code more complicated)
Step by Step Solution
There are 3 Steps involved in it
Get step-by-step solutions from verified subject matter experts
