The question is about the cross validation problem. This is the questions: (4) Cross-Validation. The above steps

The question is about the cross validation problem.

This is the questions:

Transcribed Image Text:

(4) Cross-Validation. The above steps are sufficient in many machine learning or data mining questions when both training and testing data sets are very large. However, for relatively small data sets, one may want to do further to assess the robustness of each approach. One general approach is Monte Carlo Cross Validation algorithm that splits the observed data points into training and testing subsets, and repeats the above computation B times (B = 100 say). In the context of this homework, we can combine n 1220 training data and n2 330 testing data together into a larger data set with the size of n = 1550. Then for this given larger dataset with size n = 1550 and for each loop b = 1, , B, we randomly select n 1220 as a new training subset and use the remaining n2 = 330 data as the new testing subset. Within each loop, we first build different models from "the training data of that specific loop" and then evaluate their performances on "the corresponding testing data." Therefore, for each model or method in part (3), we will obtain B values of the testing errors on B different subsets of testing data, denote by TE, for b = 1, 2,..., B. Then the "average" performances of each model can be summarized by the sample mean and sample variances of these B values: = = TE* - B 1 12 B b=1 TEb B and Var(TE) 1-2 (TE-TE-) = B = b=1 2 - Compute and compare the average" performances of each model or method mentioned in part (2). In particular, based on your results, write some paragraphs to provide a brief summary of what you discover in the cross-validation, including reporting the optimal choice of the tuning parameter k in the KNN method, and explaining how confident you are on the usefulness of your optimal choice in real-world applications. (4) Cross-Validation. The above steps are sufficient in many machine learning or data mining questions when both training and testing data sets are very large. However, for relatively small data sets, one may want to do further to assess the robustness of each approach. One general approach is Monte Carlo Cross Validation algorithm that splits the observed data points into training and testing subsets, and repeats the above computation B times (B = 100 say). In the context of this homework, we can combine n 1220 training data and n2 330 testing data together into a larger data set with the size of n = 1550. Then for this given larger dataset with size n = 1550 and for each loop b = 1, , B, we randomly select n 1220 as a new training subset and use the remaining n2 = 330 data as the new testing subset. Within each loop, we first build different models from "the training data of that specific loop" and then evaluate their performances on "the corresponding testing data." Therefore, for each model or method in part (3), we will obtain B values of the testing errors on B different subsets of testing data, denote by TE, for b = 1, 2,..., B. Then the "average" performances of each model can be summarized by the sample mean and sample variances of these B values: = = TE* - B 1 12 B b=1 TEb B and Var(TE) 1-2 (TE-TE-) = B = b=1 2 - Compute and compare the average" performances of each model or method mentioned in part (2). In particular, based on your results, write some paragraphs to provide a brief summary of what you discover in the cross-validation, including reporting the optimal choice of the tuning parameter k in the KNN method, and explaining how confident you are on the usefulness of your optimal choice in real-world applications.