4. The EM algorithm is useful in many situations, especially for parameter estimation for mixture models. Suppose we observe independent X, X2,..., X, and each X, (i=1,2,..., n) fol- lows a different distribution depending on whether it belongs to one of two groups (for example, have disease or do not have disease). Let Z, denote the indicator of whether observation X; belongs to the first group, that is, X | Z = 1 ~ f (x) and X | zi=0~ g(xi), where f(x) and g(xi) are the PDFs of the first and second group, respectively. However, we do not observe Z; and hence they are considered as "missing data" in the framework of the EM algorithm. Let p denote the probability of an observation belonging to the first group, that is, P(Z = 1) = p for i=1,2,..., n, then by the law of total probability the marginal density of Xi is fx. (x) = p f(x) + (1 - p) g(xi). The above is the density of a two-component mixture model. (a) The complete-data in this case is Y = (X, Z). Show that the density of Y is f(y)= IIp f(x)] [(1 p) g(x)]- i=1 Hint: Z; ~ Ber(p) independently. Carefully determine the conditional distri- bution of X, given Z. [3 marks] (b) In the E-step, we calculate the Q-function (that is the expected complete-data log likelihood function) which can be shown to be 12 (p, p()) = [a(*) logp + a(*) log (xi) + (1 a(*)) log(1 p) + (1 a(*) log g(x)], i=1 where a) = E(Z | x; p()). Show that Z | x, follows a Bernoulli distribution and hence find a(). [4 marks] (c) In the M-step, we maximize the Q-function to obtain an updated estimate of p. Show that on the (k+1)th iteration of the EM algorithm, the updated estimate of p is given by p(k+1); = 72 72 a(k). i=1