Hello, please modify this code:

import pyspark

sc $=$ pyspark.SparkContext$()$

def NASDAQ$($line$)$:

try:

fields $=$ line.split$(\text{'},\text{'})$

if len$($fields$)!=9$:

return False

#int$($fields$[2][$:$4])$

return True

except:

return False

def COMPANYLIST$($line$)$:

try:

fields $=$ line.split$(\text{'}\backslash$t$\text{'})$

if len$($fields$)!=5$ or $("$IPOyear$"$ in line and "Symbol" in line$)$:

return False

return True

except:

return False

#Load files and clean

nasdaq $=$ sc$.$textFile$("/$home$/$kinivera$/$BigData$/$Partial$2/$input$/$NASDAQsample$.$csv$")$

companylist $=$ sc$.$textFile$("/$home$/$kinivera$/$BigData$/$Partial$2/$input$/$companylist$.$tsv$")$

nasdaq $=$ nasdaq.filter$($NASDAQ$)$

companylist $=$ companylist.filter$($COMPANYLIST$)$

nasdaq $=$ nasdaq.map$($lambda l: $($l$.$split$(\text{'},\text{'})[1],($l$.$split$(\text{'},\text{'})[2][$:$4],$ int$($l$.$split$(\text{'},\text{'})[7]))))$ #symbol,$($date$,$n$)$

companylist $=$ companylist.map$($lambda l: $($l$.$split$("\backslash$t$")[0],$ l$.$split$("\backslash$t$")[3]))$ #symbol,sector

joined$\_$rdd $=$ nasdaq.join$($companylist$)$ #symbol,$(($date$,$n$),$sector$)$

#print$($joined$\_$rdd$.$take$(10))$

features $=$ joined$\_$rdd$.$map$($lambda row: $(($row$[1][1],$ row$[1][0][0]),$ row$[1][0][1]))$ #$($sector$,$date$),$ n

# Reduce by key $($Year$,$ Sector$)$ by adding the number of operations

sector$\_$counts $=$ features.reduceByKey$($lambda x$,$ y: x $+$ y$)$#$[(($sector$,$year$),$n$),....]$

#print$($sector$\_$counts.take$(10))$

# Find the sector with the highest number of operations for each year

max$\_$sector$\_$per$\_$year $=$ sector$\_$counts.map$($lambda x: $($x$[0][1],($x$[1],$x$[0][0])))$ #$[($year$,($n$,$sector$)),....]$

result $=$ max$\_$sector$\_$per$\_$year.reduceByKey$($lambda x$,$ y: x if x$[0]>$ y$[0]$ else y$)$ #$[($year$,($major n$,$sector$))]$

#print$($result$.$take$(10))$

#order x year

result $=$ result.sortByKey$()$

# Convert the RDD to a format suitable for saving as text

max$\_$sector$\_$per$\_$year$\_$formatted $=$ result.map$($lambda x: $($x$[1][1],"\{\},\{\}".$format$($x$[0],$ x$[1][0])))$

# Save the RDD as a text file

max$\_$sector$\_$per$\_$year$\_$formatted $=$ max$\_$sector$\_$per$\_$year$\_$formatted.coalesce$(1)$

max$\_$sector$\_$per$\_$year$\_$formatted.saveAsTextFile$("1\_$out"$)$

This was the statement given for that exercise: RDD manipulation using transformation and action operations and performance optimization using RDD are evaluated. The execution time is also evaluated

This point takes into account the Nasdaq and companylist datasets. Remember the data format is: For NASDAQ: exchange, stock symbol, date, stock opening price, stock high price, stock low price, stock closing price, stock volume

and adjusted closing price of the stock. For companylist: Symbol, Name, initial public offering year IPOyear and industry sector.

$1.$Calculate, for each year of the DataSet given for point $1,$ which sector had the greatest number of operations. The output must mention the year, the name of the sector and the overall value of operations. The result should look like:

Finance,$1996,20090342$

Pharma,$1996,12312312$

Finance,$1997,25612312$

Deliverable $1$: spark script where RDD is used to solve the problem, with the

name $1\_$topsectorperyear.py$.$ the lines must be explained within the script

code fundamentals

Deliverable $2$: Output file with the results, with the name $1\_$out.txt

Now we have to solve this statement of a Big Data exercise and data frames cannot be used.

$2.$Calculate, for each company and business sector, which company grew the most per year, also listing the percentage of growth. The results should be in a format similar to:

Finance,$1996,$ABCD,$46\%$

Finance,$1997,$VFER,$64\%$

Deliverable $3$: spark script where RDD is used to solve the problem, with the

name $2\_$topcompanypersector.py$.$ The fundamental lines of BigData$/$spark code must be explained within the script

Deliverable $4$: Output file with the results, with the name $2\_$out.txt

This must be done in a Linux virtual machine. The data companylist.tsv has headers Name, IPOyear Sector, and industry. In IPOyear some are with n$/$a and others with dates of years. The data in NADASQsample.csv has no statements.

$\backslash$table$[[$Symbol$,$Name,IPOyear,Sector,industry$],[$FLWS$,1-800$ FLOWERS.COM, Inc.,$1999,$Consumer Services,Other Specialty Stores$],[$FCTY$,1$st Century Bancshares,,n$/$a$,$Finance,Major Banks$],[$FCCY$,1$st Constitution Bancorp,n$/$a$,$Finance,Savings Institutions$],[$SRCE$,1$st Source Corporation,n$/$a$,$Finance,Major Banks$],[$FUBC$,1$st United Bancorp, Inc.,$\frac{n}{a},$Finance,Major Banks$],[$VNET$,21$Vianet Group, Inc.,$\frac{n}{a},$Technology,Computer Software:$],[$SSRX$,3$SBio Inc.,$2007,$Consumer Durables,Major Pharmaceuticals$],[$JOBS$,51$job, Inc.,$2004,$Technology,Diversified Commercial$],[$FGHT$,88lnc,\frac{n}{a},$Dublic lltiliti,$T$