Question: Q1. IDKenize function Define a function tokenize (docs, lemmatized = True, remove_stopword = True, remove_punct = True ) as follows: Take three parameters: docs: a

Q1. IDKenize function Define a function tokenize
Q1. IDKenize function Define a function tokenize (docs, lemmatized = True, remove_stopword = True, remove_punct = True ) as follows: Take three parameters: docs: a list of documents (e.g questions) lemmatized: an optional boolean parameter to indicate if tokens are lemmetized. The default value is True (i.e. tokens are lemmatized). remove_stopword: an optional bookean parameter to remove stop words. The default value is True (ie. remove stop words). Split each input document into unigrams and also clean up tokens as follows: if lemmatized is turned on, lemmatize all unigrams if remove_stopword is set to True, remove all stop words. if remove_punct is set to True, remove all punctuation tokens. remove all empty tokens and lowercase all the tokens. Return the list of tokens obtained for each document after all the processing. (Hint: you can use spacy package for this task. For reference, check https://spacy.io/api/token#attributes) def tokenize(docs, lemmatized=True, remove_stopword=True, remove_punct = True) : tokenized_docs = # add your code here return tokenized_docs Test your function with different parameter configuration and observe the differences in the resulting tokens. # For simplicity, We will test one document print (data["question"]. iloc[o] + "\ ") print (f"1. lemmatized=True, remove_stopword=False, remove_punct = True:\ \\ (tokenize(data [ 'question ']. iloc[0:1], lemmatized=True, remove_stopword False, remove_punct = True) )'n") print(f "2. lemmatized=True, remove_stopword=True, remove_punct = True:\\ n \\ {tokenize (data['question ']. iloc [0:1], lemmatized=True, remove_stopword=True, remove_punct = True))\ ") print(f "3. lemmatized=False, remove_stopword=False, remove_punct =

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Mathematics Questions!