The student should perform the operations using a small dataset without using a big data platform and then demonstrate how performance degrades with the increase in data size.

Further, the performance should be compared between big data platforms and non$-$ big data platforms using the same data size

$1.$ Scaling Material: Big Data Preprocessing

Objective: Clean, transform, and prepare a large dataset for training machine learning models or data analysis.

Tasks:

$$ Choose a large public dataset $($e$.$g$.,$ from Kaggle, UCI Machine Learning Repository, or other sources$).$

$$ Use Apache Spark or Apache Beam to handle the data.

$$ Perform data cleaning $($handling missing values, outliers, etc.$).$

$$ Transform data as needed $($normalization$,$ encoding categorical variables, adding new features etc.$).$

$$ Store the preprocessed data in a suitable format for machine learning$/$data analysis

$($e$.$g$.,$ Parquet, HDFS$).$