4. The EM algorithm is useful in many situations, especially for parameter estimation for mixture models....

Transcribed Image Text:

4. The EM algorithm is useful in many situations, especially for parameter estimation for mixture models. Suppose we observe independent X₁, X2,..., X, and each X, (i=1,2,..., n) fol- lows a different distribution depending on whether it belongs to one of two groups (for example, have disease or do not have disease). Let Z, denote the indicator of whether observation X; belongs to the first group, that is, X₁ | Z₁ = 1 ~ f (x₁) and X₁ | zi=0~ g(xi), where f(x) and g(xi) are the PDFs of the first and second group, respectively. However, we do not observe Z; and hence they are considered as "missing data" in the framework of the EM algorithm. Let p denote the probability of an observation belonging to the first group, that is, P(Z₁ = 1) = p for i=1,2,..., n, then by the law of total probability the marginal density of Xi is fx. (x) = p f(x) + (1 - p) g(xi). The above is the density of a two-component mixture model. (a) The complete-data in this case is Y = (X, Z). Show that the density of Y is f(y)= IIp f(x)] [(1 − p) g(x)]¹- i=1 Hint: Z; ~ Ber(p) independently. Carefully determine the conditional distri- bution of X, given Z₁. [3 marks] (b) In the E-step, we calculate the Q-function (that is the expected complete-data log likelihood function) which can be shown to be 12 (p, p()) = Σ [a(*) logp + a(*) log ƒ (xi) + (1 − a(*)) log(1 − p) + (1 − a(*) log g(x₁)], i=1 where a) = E(Z₁ | x; p()). Show that Z₁ | x, follows a Bernoulli distribution and hence find a(). [4 marks] (c) In the M-step, we maximize the Q-function to obtain an updated estimate of p. Show that on the (k+1)th iteration of the EM algorithm, the updated estimate of p is given by p(k+1); = 72 72 Σa(k). i=1 4. The EM algorithm is useful in many situations, especially for parameter estimation for mixture models. Suppose we observe independent X₁, X2,..., X, and each X, (i=1,2,..., n) fol- lows a different distribution depending on whether it belongs to one of two groups (for example, have disease or do not have disease). Let Z, denote the indicator of whether observation X; belongs to the first group, that is, X₁ | Z₁ = 1 ~ f (x₁) and X₁ | zi=0~ g(xi), where f(x) and g(xi) are the PDFs of the first and second group, respectively. However, we do not observe Z; and hence they are considered as "missing data" in the framework of the EM algorithm. Let p denote the probability of an observation belonging to the first group, that is, P(Z₁ = 1) = p for i=1,2,..., n, then by the law of total probability the marginal density of Xi is fx. (x) = p f(x) + (1 - p) g(xi). The above is the density of a two-component mixture model. (a) The complete-data in this case is Y = (X, Z). Show that the density of Y is f(y)= IIp f(x)] [(1 − p) g(x)]¹- i=1 Hint: Z; ~ Ber(p) independently. Carefully determine the conditional distri- bution of X, given Z₁. [3 marks] (b) In the E-step, we calculate the Q-function (that is the expected complete-data log likelihood function) which can be shown to be 12 (p, p()) = Σ [a(*) logp + a(*) log ƒ (xi) + (1 − a(*)) log(1 − p) + (1 − a(*) log g(x₁)], i=1 where a) = E(Z₁ | x; p()). Show that Z₁ | x, follows a Bernoulli distribution and hence find a(). [4 marks] (c) In the M-step, we maximize the Q-function to obtain an updated estimate of p. Show that on the (k+1)th iteration of the EM algorithm, the updated estimate of p is given by p(k+1); = 72 72 Σa(k). i=1

Answer rating: 100% (QA)

a The density of Y is fy p fx 1 p gx1 To prove this we must first determine the conditional distribu... View the full answer