Using Python:

Since I can't attach the data set, I will explain what it is:

1. News articles for each day (Jan 1, 2012 Dec 31, 2012) total 366 files
2. Every file:
   1. All the articles published in each day
   2. Each article is in a new line
   3. Each article is in json format
   4. Json Fields for each article:
      1. 'city', 'code', 'title', 'text', 'source', date'

Instructions using Python:

1. Read the data (all the files in the data directory) using the function textFile
2. Take only the text part of each article and count the frequency of all the words (convert the text into lowercase)
3. Remove (Filter) any word whose frequency is less than 10
4. Report the following:
   1. Total size of the output data (after the filtering)
   2. Frequency of the following words congress, london, washington, football
   3. The word with maximum frequency for each month (hint: to read only a months articles you can use *. E.g. for February 2012-02* represents all files starting with 2012-02,i.e. files belonging to Feb)
   4. List of words appeared on 2012-09-01 but not on 2012-08-01
   5. Frequency of the word monsoon for all months