You have been provided with a starter notebook that reads a collection of tweets and a collection

Fantastic news! We've Found the answer you've been seeking!

Question:

You have been provided with a starter notebook that reads a collection of tweets and a collection of news articles

You need to determine which news articles (news_df) are similar to each other and which tweets (tweets_df) are more similar to each other. In order to accomplish this you need to create n-grams and compare the similarity of the text using Jaccard distance.

Additional instructions:

To get quality results apply appropriate text cleaning methods
Your submission must be a Python Notebook (ipynb)
Use 'Markdown' in the cell to document your answers / provide comments as needed
Visualize your results instead of writing about them. Remember, a picture is worth a thousand words

Your final submission must include the following:

Which news articles / tweets were similar and which ones were dissimilar?
A brief write-up explaining why and how you chose "n" for you analysis (for n-grams)
Was the "n" identical or different for articles vs. tweets and why
Visualize the selection of "n"
For the news articles, please explain why you chose text or title or both text + title combined
Include all of your program codes (creating n-grams from text as well as selecting the "n" for analysis)

Here's my code so far:

import pandas as pd

pd.set_option('display.max_rows', 100)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', 500)

news_path = 'https://storage.googleapis.com/msca-bdp-data-open/news/nlp_a_3_news.json'
news_df = pd.read_json(news_path, orient='records', lines=True)

print(f'Sample contains {news_df.shape[0]:,.0f} news articles')
news_df.head(2)

tweets_path = 'https://storage.googleapis.com/msca-bdp-data-open/tweets/nlp_a_3_tweets.json'
tweets_df = pd.read_json(tweets_path, orient='records', lines=True)
print(f'Sample contains {tweets_df.shape[0]:,.0f} tweets')
tweets_df.head(2)