Consider $3$ files warc.csv$,$ wat.csv and we$.$csv without headers.

Consider a warc.csv file related data. An indicative line is:

$2017-03-22T22$:$13$:$57Z,$ urn, response, $40802,213\mathrm{.}155\mathrm{.}18\mathrm{.}48$

$,http$:coco, Apache, $html$

Columns in order: first the warc date, the warc record id$,$ the warc type $($e$.$g$.$ metadata,

response, etc$),$ the content length, the public IP address, the target URL, the server running

the site $($eg apache, nginx, etc$),$ and finally the overall content of the page with the entire

HTML DOM.

Consider a wat.csv file related data. An indicative line is:

urn:uuid, $1053,$ http:coco

In order the columns are: first the warc record id$,$ the content length of the metadata, and

finally the target URL $($it can be different from the target URL of the warc data$).$

Consider a wet.csv file related data. An indicative line is:

urn:uuid, "extracted plaintext"

In order the columns are: first the warc record id and then the extracted plaintext from the

url $($can be in ascii$).$

Using RDDs write a Pyhton code to answer the following.

Task $1$:

Find the most popular target URL $($eg$,$ the record target URL that can be found in the HTML

DOM of another record.

Tips: You will need to join datasets to get the desired result. For this query you will need to

filter out the records that have null values. You should first find for each warc record what

its target URL is and what URLs are in the HTML DOM, so you get an intermediate result:

targetURL $->$ list$($urls in html dom$).$ For the URLs you could simplify them and keep a

simpler format$/$subdomain to get even more results.

Remember to restart the Spark cluster before each measurement, to avoid hot caches, or

you can clear the cache.

Task $2$:

Perform Task $1$ using DataFrames$/$Spark SQL and parquet file