Problem 4

Parameters to be tuned for XGBoost: 1. n_estimators 2. max_depth 3. lambda 4. learning_rate 5. missing 6. objective Parameters to be tuned for SVM: 1. kernel_type 2. gamma 3. C Parameters to be tuned for Random Forests: 1. n_estimators 2. bootstrap 3. max_depth 4. min_impurity_decrease 5. min_samples_leafExperiment with non-linear classifiers Problem 4 (40 points) For this problem, you will need to learn to use software libraries for at least two of the following non-linear classifier types: . Boosted Decision Trees (i.e., boosting with decision trees as weak learner) . Random Forests . Support Vector Machines with Gaussian Kernel All of these are available in acikit-learn, although you may also use other external libraries (e.g., XGBoost ' for boosted decision trees and LibSVM for SVMs). You are welcome to implement learning algorithms for these classifiers yourself, but this is neither required nor recommended. Pick two different types of non-linear classifiers from above for classification of Adult dataset. You can the download the data from aga in libSVM data repository. The aga data set comes with two files: the training data file a9a with 32,561 samples each with 123 features, and aga. t with 16,281 test samples. Note that a9a data is in LibSVM for- mat. In this format, each line takes the form : : ..... This format is especially suitable for sparse datasets. Note that scikit-learn includes utility functions (e.g., load_avmlight_file in example code below) for loading datasets in the LibSVM format. For each of learning algorithms, you will need to set various hyperparameters (e.g., the type of kernel and regularization parameter for SVM, tree method, max depth, number of weak classifiers, etc for XGBoost, number of estimators and min impurity decrease for Random Forests). Often there are defaults that make a good starting point, but you may need to adjust at least some of them to get good performance. Use hold-out validation or K-fold cross-validation to do this (scikit-learn has nice features to accomplish this, e.g., you may use train_test_split to split data into train and test data and aklearn . model_selection for K-fold cross validation). Do not make any hyperparameter choices (or any other similar choices) based on the test set! You should only compute the test error rates after you have settled on hyperparameter settings and trained your two final classifiers. What to submit (in PDF file and Jupyter/python codes): 1. Names of the two classifiers you opt to learn and a brief description of each algorithm and how it works. 2. Description of your training methodology, with enough details so that another ma- chine learning enthusiast can reproduce the your results. You need to submit all the codes (python and Jupyter notebooks) to reproduce your code. Please use pre- fix Problem4* . py where you need to replace "* with the name of non-linear classifier for your coding files. 3. The list of hyperparameters and breif description of each hyperparameter you tuned in training, their default values, and the final hyperparameter settings you use to get the best result. 3 A simple blog post on how to use XCBoost please check this. 4. Training error rates, hold-out or cross-validation error rates, and test error rates for your two final classifiers. You are also encouraged to report other settings you tried with the accuracy it achieved (please make a table with a column with each hyperparamter and accuracy of configuration of parameters. 5. Please do your best to obtain the best achievable accuracy for each classifier on given dataset. Note: The amount of effort you put on tuning the parameters will be deter- mined based on the discrepancy between the accuracy you get and the best achievable accuracy on aga data for each algorithm