Consider the following approach for testing whether a classifier A beats another classifier B. Let N be the size of a given data set, pA be the accuracy of classifier A, pB be the accuracy of classifier B, and p = (pA + pB)/2 be the average accuracy for both classifiers. To test whether classifier A is significantly better than B, the following Z-statistic is used:

Classifier A is assumed to be better than classifier B if Z > 1.96. Table 4.3 compares the accuracies of three different classifiers, decision tree classifiers, naïve Bayes classifiers, and support vector machines, on various data sets. (The latter two classifiers are described in Chapter 5.)

PA-PB 2p(1-p) 2 84 70 96 84 85 76 74 59 83 87 82 88 96 92 73 86 76 76 98 74 98 96 979 21 7 7 7 8 9 8 7 7 7 8 8 8 8 8 95 9 3 3 76 69 70 45 6 (9-9 5 5 4 4 8 0 90 2 00 4 3 7 1 591284 0 68888977 4831 7 5 20 8 69 5 9 3 0 8 000 21 2 5 6 5 5 57 200 Las 750 886 17 101 221331 cDGGHHH LPSTVW