As a senior Data Engineer, your role extends beyond applying machine learning algorithms; it includes

optimizing models, ensuring data privacy, and making strategic decisions based on data insights. This

assessment is designed to deepen your expertise in machine learning through complex real$-$world dataset

applications.

Instructions

This assignment requires applying advanced classification algorithms to the MNIST dataset and conducting an in$-$

depth regression analysis on the California housing dataset. Each task must demonstrate not only technical

proficiency but also strategic thinking and ethical considerations.

Tasks

$1.$ Advanced Dataset Preparation

$$ Split the MNIST dataset into training and testing sets using scikit$-$learn functions or your own

custom function. Include detailed comments to explain your process.

$$ Conditions:

$$ Use your first name for the training set and your last name for the testing set variable

names. For instance, If your name is john doe, use john$\_$train and doe$\_$test as your

variable names.

$$ Use the last two digits of your student ID as the random$\_$state for any function that requires

it$.$ For instance, the value for the random$\_$state $=[$Last two digits of your student ID$]$

$$ Utilize advanced preprocessing techniques to enhance model performance, such as feature scaling

and dimensionality reduction where appropriate.

$2.$ Advanced kNN Classifier

$$ Set k$=10$ and utilize the kNN classifier from scikit$-$learn, providing a detailed explanation of the

function parameters and their implications.

$$ Evaluate the model using advanced metrics. Discuss the rationale behind choosing specific metrics

and their implications on the model evaluation.

$$ Conduct an in$-$depth analysis of varying k values, including a statistical test to determine if changes

in performance are significant.

$3.$ SVM Classifier with Parameter Optimization

$$ Apply an SVM classifier using both linear and non$-$linear kernels. Experiment with feature

engineering techniques to improve model accuracy.

$$ Discuss the kernel trick and its impact on the computational complexity and performance of the

model