Machine Learning

Exercise 3.6 [Cross-entropy error measure) (a) More generally, if we are learning from +1 data to predict a noisy target Py | x) with candidate hypothesis h, show that the maximum likelihood method reduces to the task of finding h that minimizes N 1 1 Ein(w) = [yn = +1] In + [yn h(xn) = 1] In 1 h(Xn) n=1 (b) For the case h(x) = 0(w"x), argue that minimizing the in sample error in part (a) is equivalent to minimizing the one in (3.9). For two probability distributions {p, 1 p} and {q,1 q} with binary out- comes, the cross entropy (from information theory) is 1 plog = + (1 - p) log 9 1-4 The in sample error in part (a) corresponds to a cross entropy error measure on the data point (Xn, Yn), with p= [yn = +1] and q = h(xn). N Ein(w) In In (1+e+vwx. N n=1 (3.9) Exercise 3.6 [Cross-entropy error measure) (a) More generally, if we are learning from +1 data to predict a noisy target Py | x) with candidate hypothesis h, show that the maximum likelihood method reduces to the task of finding h that minimizes N 1 1 Ein(w) = [yn = +1] In + [yn h(xn) = 1] In 1 h(Xn) n=1 (b) For the case h(x) = 0(w"x), argue that minimizing the in sample error in part (a) is equivalent to minimizing the one in (3.9). For two probability distributions {p, 1 p} and {q,1 q} with binary out- comes, the cross entropy (from information theory) is 1 plog = + (1 - p) log 9 1-4 The in sample error in part (a) corresponds to a cross entropy error measure on the data point (Xn, Yn), with p= [yn = +1] and q = h(xn). N Ein(w) In In (1+e+vwx. N n=1 (3.9)