The Leave-One-Out Cross-Validation, or LOOCV, procedure is used to obtain a reliable and unbiased estimate of...

Transcribed Image Text:

The Leave-One-Out Cross-Validation, or LOOCV, procedure is used to obtain a reliable and unbiased estimate of model performance. In this approach, the data set is divided into n subsets where n is the number of instances in the dataset. Here, each subset only contains one point. Each time, one of the n subsets is used as the test set and the other n-1 subsets are put together to form a training set. These training examples can be used to construct a predictive model. Then this model is asked to predict the output value for the data that is left out as the test point (the model has never seen this test point before). We can then evalute the performance of the model on the test example. This procedure is repeated n times until all the examples are considered as test data at some iteration. Then the average error across all n trials is computed. The advantage of this method is that it matters less important how the data gets divided into training and testing data. Every data point gets to be in a test set exactly once, and gets to be in a training set n-1 times. The disadvantage of this method is that the training algorithm has to be rerun from scratch n times, which means it takes n times as much computation to make an evaluation. We want to practice this approach to estimate the error rate of a k-nearest neighbors method. Let's consider a 3-nearest neighbor classifier using Manhattan distance metric on a binary classification task. What would be the LOOCV error this model on the following dataset. 10 9 8 N 6 5 4 3₂2 2 1 0 1 2 3 5 6 The Leave-One-Out Cross-Validation, or LOOCV, procedure is used to obtain a reliable and unbiased estimate of model performance. In this approach, the data set is divided into n subsets where n is the number of instances in the dataset. Here, each subset only contains one point. Each time, one of the n subsets is used as the test set and the other n-1 subsets are put together to form a training set. These training examples can be used to construct a predictive model. Then this model is asked to predict the output value for the data that is left out as the test point (the model has never seen this test point before). We can then evalute the performance of the model on the test example. This procedure is repeated n times until all the examples are considered as test data at some iteration. Then the average error across all n trials is computed. The advantage of this method is that it matters less important how the data gets divided into training and testing data. Every data point gets to be in a test set exactly once, and gets to be in a training set n-1 times. The disadvantage of this method is that the training algorithm has to be rerun from scratch n times, which means it takes n times as much computation to make an evaluation. We want to practice this approach to estimate the error rate of a k-nearest neighbors method. Let's consider a 3-nearest neighbor classifier using Manhattan distance metric on a binary classification task. What would be the LOOCV error this model on the following dataset. 10 9 8 N 6 5 4 3₂2 2 1 0 1 2 3 5 6