D Bookmark this page

Homework due Aug $7,202417$:$29$ IST

Consider the following mixture of two Gaussians:

$p(x$;$)={}_{1}N(x$;${}_1,{}_1^2)+{}_{2}N(x$;${}_2,{}_2^2)$

This mixture has parameters $=\{{}_1,{}_2,{}_1,{}_2,{}_1^2,{}_2^2\}.$ They correspond to the mixing proportions,

means, and variances of each Gaussian. We initialize $$ as ${}_0=\{0\mathrm{.}5,0\mathrm{.}5,6,7,1,4\}.$

We have a dataset $D$ with the following samples of $x$ : $x^{(0)}=-1,x^{(1)}=0,x^{(2)}=4,x^{(3)}=5,x^{(4)}=6.$

We want to set our parameters $$ such that the data log$-$likelihood $l(D$;$)$ is maximized:

$argma{x}_{}{\displaystyle \underset{i=0}{\overset{4}{}}}logp(x^{(i)}$;$)$

Recall that we can do this with the EM algorithm. The algorithm optimizes a lower bound on the log$-$likelihood,

thus iteratively pushing the data likelihood upwards. The iterative algorithm is specified by two steps applied

successively:

E$-$step: infer component assignments from current ${}_0=($complete the data$)$

$p(y=k|x^{(i)})$:$=p(y=k|x^{(i)}$;${}_0),$ for $k=1,2,$ and $i=0,$dots,$4.$

M$-$step: maximize the expected log$-$likelihood

tilde$(l)(D$;$)$:$={\displaystyle \underset{i}{\overset{?}{}}}{\displaystyle \underset{k}{\overset{?}{}}}p(y=k|x^{(i)})log((p\frac{x^{(i)},y}{p(y}=k|x^{(i)})=k|x^{(i)})=k$;$)$

with respect to $$ while keeping $p(y=k|x^{(i)})$ fixed.

To see why this optimizes a lower bound, consider the following inequality:

$logp(x$;$)=log{\displaystyle \underset{y}{\overset{?}{}}}p(x,y$;$)$

$=log{\displaystyle \underset{y}{\overset{?}{}}}q(y|x)\frac{p(x,y\text{;})}{q(y|x)}$

$=log{E}_{yq(y|x)}[\frac{p(x,y\text{;})}{q(y|x)}]$

$E_{yq(y|x)}[log(\frac{p(x,y\text{;})}{q(y|x)})]$

$={\displaystyle \underset{y}{\overset{?}{}}}q(y|x)log(\frac{p(x,y\text{;})}{q(y|x)})$