In this project you will develop tools for performing sentiment analysis on a database of tweets from across the country. When the project is complete you should be able to estimate the sentiment of tweets filtered by content.

There are 4 files provided here:

all_tweets.txt is the large collection of tweets

some_tweets.txt is a subset of all_tweets that's more manageable to prototype on

sentiments.csv a csv with word sentiment values

zips.csv (not required, see below)

We will go over the format of each of these files in class.

Tweets: We will represent a tweet using a Python dictionary with the following entries:

text: a string, the text of the tweet all in lowercase

time: a datetime object, date and time of the tweet

latitude: a float, the latitude of the tweet's location

longitude: a float, the longitude of the tweet's location

Problem 1 Write a function called make_tweets that takes as input a file name and returns a list of dictionaries. Each dictionary corresponds to a tweet.

In [ ]:

def make_tweets(filename): #your code here 

Problem 2 Write a function add_sentiment to determine the sentiment of each tweet by taking the average sentiment over all of the words in the tweet. The function should return a new list of tweets where each tweet has a new key 'sentiment' with a numeric value between -1 and 1, or None representing the sentiment of the tweet. Note: words without a sentiment do not have sentiment 0. Your function should take as input a list of tweets (dictionaries) together with the name of the sentiment file. Be careful that your function does not alter the original list (no side effects!)

In [ ]:

def add_sentiment(tweets,filename): #your code here 

Problem 3 Write a function called tweet_filter that will return a new list of tweets filtered by the content of the tweet text. The input for this function should be a list of tweets and a list of words (strings). The function should return a list of tweets that each include all of the words in the word list ignoring case and punctuation. Note: Since you are not changing the tweets, as long as the returned list is new, you don't have to worry about side-effects on the tweets here.

In [ ]:

def tweet_filter(tweets, words): #your code here 

Problem 4 Use your work above and below to answer the following questions:

What is the average sentiment of tweets containing the word 'beer'

What is the average sentiment of tweets containing the word 'coffee'

Consider the average sentiment of the tweets containing at least one of the words 'beer', 'movie', coffee', 'work'. Which word leads to a list of tweets with the lowest average sentiment?

In [ ]:

# Include the code you wrote for Problem 4 here 

Write the answers to problem 4 here:

Average sentiment:

Average sentiment:

Lowest average sentiment: