Question: Q2. Tokenization function: Write a function that gets a sentence or a text as its input parameter, breaks it into separate words and returns a

 Q2. Tokenization function: Write a function that gets a sentence or

Q2. Tokenization function: Write a function that gets a sentence or a text as its input parameter, breaks it into separate words and returns a list of the words extracted from the text. For example, let the function's name be "tokenize", then if the following sentence is passed to the function: "this is my first class, I love my class". The returning result must be: [this, is, my, first, class, , I, love, my, class In other words, if we call the method as show below: tokenize("this is my first class, I love my class") the following result will be returned: ['this', 'is', 'my', 'first', 'class,', 'I', 'love', 'my', 'class'] Every single element of the list is an individual word extracted from the text. All Words in the text are separated based on spaces between them (if there is a space between two consecutive words, they must be separated and listed as two different words). As a result, "data set" will be treated as two separate words rather than a single word. Note: for the sake of simplicity we assume that there is exactly one space between every two consecutive words (words in the sentence are separated by exactly one space, because if there is more than one space between each pair of words then It makes the code more complicated)

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!