Question: Consider the Hitters dataset from the package ISLR. You will use the following code for preprocessing and use the hitters.comp as your data. 1 library

Consider the "Hitters" dataset from the package "ISLR". You will use the following code for preprocessing and use the "hitters.comp" as your data. 1 library (ISLR) 2 hitters.comp = na.omit(Hitters) 3 hitters.comp$Salary = log(hitters.comp$Salary) 4 set. seed (11) 5 n = nrow(hitters.comp) 6 test.id = sample(1:n, round(0.4%n)) Line 2 removes those rows with missing values. Line 3 does a log transformation on the variable "Salary". Line 6 creates a list "test.id" that contains the indexes of rows for test dataset; that is, those rows whose indexes are not in test.id will be used as training data. The task is to predict log(Salary) using other covariates. (a). Run the linear regression on the training data using all the covariates, and report the mean square error on the test data. (b). Perform 5-fold cross-validation (CV) for the ridge regression on the training data using the function "cv.glmnet", and plot the CV errors. Further, report the mean square error of the ridge regression on the test data with chosen to be the one minimizing the average CV error. Note that in CV, use the following sequence for 1: exp(seq(-10, 10, length=1000)). (c). Run the Lasso on the training data using the function "gimnet", and plot the solution path, i.e., the values of the estimated coefficients against log(^). (d). Perform 10-fold CV for the Lasso on the training data using the function "cv.glmnet", and plot the CV errors. Further, report the number of covariates (excluding intercept) selected by Lasso with a chosen by the "one-standard error" rule. Finally, report the mean square error of the Lasso on the test data with chosen to be the one minimizing the average CV error. Note that in CV, use the same sequence for as in (b). Hint: both glmnet and cv.glmnet function require a design matrix as the first argument. Since some covariates are categorical, you may need to use the "model.matrix" function
Step by Step Solution
There are 3 Steps involved in it
Get step-by-step solutions from verified subject matter experts
