Data description

News articles for each day $($Jan $1,2012$ Dec $31,2012)$ total $366$ files

Every file:

All the articles published in each day

Each article is in a new line

Each article is in JSON format

JSON Fields for each article:

'city', 'code', 'title', 'text', 'source', $$date$\text{'}$

Problem:

Complete the following using spark:

Read the data $($all the files in the $$data$$ directory$)$ using the function textFile

Take only the $$text$$ part of each article and count the frequency of all the words $($convert the text into lowercase$)[2$ point$]$

Remove $($Filter$)$ any word whose frequency is less than $10[2$ points$]$

Report the following:

The total size of the output data $($after the filtering$)[2$ points$]$

The frequency of the following words $$ congress, london, washington, football $[2$ points$]$

The word with maximum frequency for each month $($hint: to read only a month$$s articles, you can use $*.$ E$.$g$.,$ for February $2012-02*$ represents all files starting with $2012-02,$i$.$e$.,$ files belonging to Feb$)[4$ points$]$

The list of words that appeared on $2012-09-01$ but not on $2012-08-01[4$ points$]$

The frequency of the word $$monsoon$$ for all months $[4$ points$]$