Consider $3$ files warc.csv$,$ wat.csv and we$.$csv$.$

Consider a warc.csv file related data. An indicative line is:

$2017-03-22$T$22$:$13$:$57$Z$,$ urn:coco, response, $40802,$

$213\mathrm{.}155\mathrm{.}18\mathrm{.}48,$ http:coco, Apache, $\text{'}$html$\text{'}$

Columns in order: first the warc date, the warc record id$,$ the warc type $($e$.$g$.$ metadata,

response, etc$),$ the content length, the public IP address, the target URL, the server running

the site $($eg apache, nginx, etc$),$ and finally the overall content of the page with the entire

HTML DOM.

Consider a wat.csv file related data. An indicative line is:

urn:uuid, $1053,$ http:coco

In order the columns are: first the warc record id$,$ the content length of the metadata, and

finally the target URL $($it can be different from the target URL of the warc data$).$

Consider a wet.csv file related data. An indicative line is:

urn:uuid, "extracted plaintext"

In order the columns are: first the warc record id and then the extracted plaintext from the

url $($can be in ascii$).$

Using RDDs write a Pyhton code to answer the following.

Task $1$:

For each server used, find the average content length of warc records as well as the average

content length of their metadata $($print the top $5$ servers with the largest content size$).$

Tips: You will need to join datasets to get the desired result.

For this query you will need to filter out the records that have null values.

Remember to restart the Spark cluster before each measurement, to avoid hot caches, or

you can clear the cache.

Perform Task $1$ using DataFrames$/$Spark SQL and parquet file

Task $2$:

Please give instruction on how to measure the execution time for Task$1$:

Map$/$Reduce$-$RDD API

Spark SQL to CSV files $($you need to define the schema of each file$)$

Spark SQL to parquet file