Question: Complete the following using spark: 1 . Read the data ( all the files in the data directory ) using the function textFile 2 .

Complete the following using spark:
1. Read the data (all the files in the data directory) using the function textFile
2. Take only the text part of each article and count the frequency of all the words (convert the text into lowercase)[2 point]
3. Remove (Filter) any word whose frequency is less than 10[2 points]
4. Report the following:
1. The total size of the output data (after the filtering)[2 points]
2. The frequency of the following words congress, london, washington, football [2 points]
3. The word with maximum frequency for each month (hint: to read only a months articles, you can use *. E.g., for February 2012-02* represents all files starting with 2012-02,i.e., files belonging to Feb)[4 points]
4. The list of words that appeared on 2012-09-01 but not on 2012-08-01[4 points]
5. The frequency of the word monsoon for all months [4 points]

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Programming Questions!