Question: i need answer step by step inWEKA Part I: Algorithm Comparison 1. In this lab you will investigate the difference in model performance using statistical
i need answer step by step inWEKA
Part I: Algorithm Comparison 1. In this lab you will investigate the difference in model performance using statistical significance testing. We will compare four models (decision tree J48, 3-Nearest-Neighbor and SVM) on two different data sets (diabetes. arff and breast-cancer, arff), and perform a pairwise comparison of the models on each data set (You can do a total of six paired experiments separately or run everything at the same time). 2. Choose 10 folds cross-validation as your experiment type and repeat 5 times on each pair. 3. For both data sets, compare the performance of all three algorithms using a paired t-test. For each model, describe parameter settings/design decisions you make in acquiring your data (so that your experiments are replicable). 4. You can collect accuracy estimates using the Experimenter in WEKA, dumping the results to a CSV file and using the appropriate column of the file. You will need to implement the paired permutation test yourself. 5. Does any one of the algorithms work significantly differently on either one of the two datasets from another algorithm? Report your findings. You should use sereenshots, calculations and analysis to support your conclusions. 6. For each pair of algorithms that you find to perform significantly differently, calculate the p-value of the paired t-test to support your finding. Part I: Algorithm Comparison 1. In this lab you will investigate the difference in model performance using statistical significance testing. We will compare four models (decision tree J48, 3-Nearest-Neighbor and SVM) on two different data sets (diabetes. arff and breast-cancer, arff), and perform a pairwise comparison of the models on each data set (You can do a total of six paired experiments separately or run everything at the same time). 2. Choose 10 folds cross-validation as your experiment type and repeat 5 times on each pair. 3. For both data sets, compare the performance of all three algorithms using a paired t-test. For each model, describe parameter settings/design decisions you make in acquiring your data (so that your experiments are replicable). 4. You can collect accuracy estimates using the Experimenter in WEKA, dumping the results to a CSV file and using the appropriate column of the file. You will need to implement the paired permutation test yourself. 5. Does any one of the algorithms work significantly differently on either one of the two datasets from another algorithm? Report your findings. You should use sereenshots, calculations and analysis to support your conclusions. 6. For each pair of algorithms that you find to perform significantly differently, calculate the p-value of the paired t-test to support your finding
Step by Step Solution
There are 3 Steps involved in it
1 Expert Approved Answer
Step: 1 Unlock
Question Has Been Solved by an Expert!
Get step-by-step solutions from verified subject matter experts
Step: 2 Unlock
Step: 3 Unlock
