Question: please solve using file path only ...others i can do as file is not accessible due to tech issue in server Complete the python program

please solve using file path only ...others i can do as file is not accessible due to tech issue in server

Complete the python program file assignment2.py using spark to do the following:

  1. Read the data (all the files in the data directory) using the function textFile
  2. Take only the text part of each article and count the frequency of all the words (convert the text into lowercase)
  3. Remove (Filter) any word whose frequency is less than 10
  4. Report the following:
    1. Total size of the output data (after the filtering) [1 point]
    2. Frequency of the following words congress, london, washington, football [1 point]
    3. The word with maximum frequency for each month (hint: to read only a months articles you can use *. E.g. for February 2012-02* represents all files starting with 2012-02,i.e. files belonging to Feb) [3 points]
    4. List of words appeared on 2012-09-01 but not on 2012-08-01 [3 points]
    5. Frequency of the word monsoon for all months [2 points]

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Databases Questions!