Q$1.$ Corpus$-$Based Chatbot: Code $[10]$

In this task, you will learn to implement the corpus$-$based chatbot using a retrieval approach.

$1.$ Use the NPS chat dataset from NLTK and load the sentences using nltk$.$ corpus.nps$\_$chat.posts$().$

$2.$ From the sentences, discard all the sentences which are questions. You may use re package to find all the sentences starting with words like 'what', 'why', 'when', 'where', $\text{'}$is$\text{'},$ 'how', $\text{'}$do$\text{'},$ 'does', 'which', 'are', 'could', 'would', 'should', 'has', 'have', 'whom', 'whose', and 'don't'. Also, remove the sentences with lengths less than or equal to $4$ unless the sentence contains words like 'hello', $\text{'}$hi$\text{'},$ 'greetings', 'what's up$\text{'},$ and 'hey'.

$3.$ Using the TF$-$IDF vectorization implemented in assignment $4,$ calculate the TF$-$IDF vectors for the sentences.

$4.$ Write a method to calculate the cosine similarity of a vector with all the documents and return the document with the highest similarity score.

$5.$ Build a chatbot by iteratively taking the user input, converting it to a TF$-$IDF vector, and finding and displaying the most similar document using the cosine similarity of the vector with all documents in the corpus.

$6.$ Written: Explore the chatbot with your own input. Specify $10$ inputs you provided and output generated from the chatbot and give your analysis based on the following criteria:

$-$ Engagingness: On a scale of $1-5,$ how much did you enjoy talking to the chatbot?

$-$ Making sense: On the following scale $[1-4],$ how often did the chatbot say something which did NOT make sense?

$($a$)$ Never made any sense $(1)$

$($b$)$ Most responses didn't make sense $(2)$

$($c$)$ Some responses didn't make sense $(3)$

$($d$)$ Everything made perfect sense $(4)$

$-$ Avoiding Repetition: On the following scale $[1-3],$ how repetitive was the chatbot?

$($a$)$ Repeated themselves over and over $(1)$

$($b$)$ Sometimes said the same thing twice $(2)$

$($c$)$ Always said something new $(3)$

$-$ Fluency: On a scale of $1-5,$ how fluent were the responses of the chatbot $($grammatical correctness, clarity, readability$)?$