Evaluate Classifier

ticular about how her clothes fade after washing. She notices that her predictions on

whether a piece of clothing will fade or not are often incorrect. To improve her predic$-$

tions, Alice decides to create her own decision tree to help her determine whether her

clothes will fade $($TRUE if the clothes fade; FALSE if they don't$).$ The decision tree

Alice created is shown in the Figure below.

Your task is to determine whether to prune the given decision tree. Additionally, you will

use the provided test dataset in $"$hw$2\_$q$3.$csv$"$ to assess the effectiveness of the resulting

decision trees

Figure $1$: The Color Prediction decision tree.

$($a$)(10$ points$)$ Post$-$pruning based on optimistic errors

$(3$ points$)$ Calculate the optimistic errors before splitting and after splitting

ii$.(2$ points$)$ Based upon the optimistic errors, would the subtree be pruned or

retained? If it is pruned, draw the resulting decision tree and use it for the

next question; otherwise, use the original decision tree shown in Figure $1$ for

the next question

iii. $(5$ points$)$ Use the decision tree from $($a$).$ii above to classify the test dataset

$($hw$2\_$q$3.$csv$).$ Report its performance on the following five evaluation metrics:

Accuracy, Recall $($Sensitivity$),$ Precision, Specificity, and F$1$ Measure.

$($b$)(10$ points$)$ Post$-$pruning based on pessimistic errors. When calculating pes$-$

simistic errors, each leaf node will add a factor of $2$ to the error.

i$.(3$ points$)$ Calculate the pessimistic errors before splitting and after splitting

using the attribute Act respectively.

ii$.(2$ points$)$ Based on the pessimistic errors, would the subtree be pruned or

retained? If it is pruned, draw the resulting decision tree and use it for the

next question; otherwise, use the original decision tree shown in Figure $1$ for

the next question.

iii. $(5$ points$)$ Use the decision tree from $($b$).$ii above to classify the test dataset

$($hw$2\_$q$3.$csv$).$ Report its corresponding five evaluation metrics: Accuracy, Re$-$

call$($Sensitivity$),$ Precision, Specificity, and F$1$ Measure.

$($c$)(10$ points$)$ We will compare the performance of the decision trees from $($a$).$ii and

from $($b$).$ii using the test dataset $($hw$2\_$q$3.$csv$).$ For the task of predicting if Alice

will get inflated, which of the five evaluation metrics: Accuracy, Recall$($Sensitivity$),$

Precision, Specificity, and F$1$ Measure, are the most important? Based on your

selected evaluation metrics, which decision tree, $($a$).$ii or $($b$).$ii$,$ is better for this

task? Justify your answers.