Ex2.

Given a training set D, derive the maximum likelihood estimates of the naive Bayes.

Exercise 1 [5 points]. This problem reviews basic concepts from probability. a) [1 point]. A biased die has the following probabilities of landing on each face: face 1 2 3 4 5 6 P(face) 1 . 1 .2 .2 .4 0 I win if the die shows even. What is the probability that I win? Is this better or worse than a fair die (i.e., a die with equal probabilities for each face)? b) [1 point]. Recall that the expected value E[X] for a random variable X is E[X] = EP(X = 1) I, TEA where \\' is the set of values X may take on. Similarly, the expected value of any function f of random variable X is ES(X)] = _P(X = x) f(x). TEA Now consider the function below, which we call the "indicator function" 1 if X = a I[X = ] jo ixfa Let X be a random variable which takes on the values 3, 8 or 9 with probabilities pa, ps and po respectively. Calculate E[I [X = 8]].Exercise 2 [5 points]. Given a training set D = {(x),y) ), i = 1, ..., M}, where I() E RN and y() E {1, 2,..., C}, derive the maximum likelihood estimates of the naive Bayes for real valued r." modeled with a Laplacian distribution, i.e., p(x; ly = c) = H exp ; - Hill 20 jlcc) [2 points]. Recall the following definitions: . Entropy: H(X) = - Drex P(X = x) log2 p(X = x) = -E[log, p(X)] . Joint entropy: H(X, Y) = -ExEX Drey P(X = x, Y = y/) log2 p(X = x, Y =y) = -E[log, p(X, Y)] . Conditional entropy: H(Y X) = -Drex Dyeyp(X - x, Y = v) log2 p(Y = yX = x) = -E[log2 p(Y|X")] . Mutual information: I(X; Y) = Drex DreyP(X = x, Y = y) log, PASSYET 2 p(X=D)p(Y=y) Using the definitions of the entropy, joint entropy, and conditional entropy, prove the following chain rule for the entropy: H(X, Y ) = H(Y) + H(X|Y). d) [1 point]. Recall that two random variables X and Y are independent if for all re X and all y e ), p(X = x,Y = y) = p(X =x)p(Y =y). If variables X and Y are independent, is I(X; Y) = 0? If yes, prove it. If no, give a counter example