To evaluate the true error rate of a machine learning approach (binary decision), Bob suggests using the 200 available instances (or examples) for both the training and the test. So, he said we have 200 examples for training and 200 to test the system. Thus, we use all the available information both for training and for testing. Alice said that this is unfair because Bob will use the same instances for both the training and the test stage. Alice proposes to use a hold-out approach, using the first 150 examples for training the system and the rest (50 examples) to evaluate it. To this point, Bob said that Alice does not use all the available information for the training and the test phase. Thus he said using less examples in the training induces a less perfect system. a) Between Alices and Bobs methodology, which one do you prefer and why (justify your answer).

b) Can you provide another evaluation methodology that can (nearly) satisfy both desiderata (Alice wants a fair evaluation and Bob wants to use all the 200 examples for both the training and testing)? Explain why your answer, if any, satisfy both conditions. Otherwise, explain why it is impossible to satisfy both constraints. c) After doing the evaluation, Bob achieves the following result. From the 200 examples, the classifier correctly classifies 90 instances with the correct label true. From this set of instances having the true label, 10 have been classified as false.

Compute the accuracy, the precision and recall of this solution.

\begin{tabular}{|c|c|c|c|} \hline & \multicolumn{3}{|c|}{ True state of Nature } \\ \hline Predicted & True & False & Total \\ \hline True & 90 & 5 & 95 \\ \hline False & 10 & 95 & 105 \\ \hline Total & 100 & 100 & 200 \\ \hline \end{tabular}