Gaussian Mixture Model: Definitions

A Gaussian Mixture Model $($GMM$),$ which is a generative model for data $n{R}^d,$ is defined by the following

set of parameters:

$K$ : Number of mixture components

A $d-$dimensional Gaussian $N({}^{(j)},{}_j^2)$ for every $j=1,$dots,$K$

$p_1,$dots,$p_K$ : Mixture weights

The parameters of a $K-$component GMM can be collectively represented as

$=\{p_1,$dots,$p_K,{}^{(1)},$dots,${}^{(K)},{}_1^2,$dots,${}_K^2\}.$ Note that we have assumed the same variance ${}_j^2$ across

all components of the $j^{th}$ Gaussian mixture component for $j=1,$dots,$K.$ Further, every Gaussian component

is assumed to have a diagonal covariance matrix. These are two assumptions that are made only for simplicity

and the methodology presented can be extended to the setting of a general covariance matrix. Also, note that

${}^{(j)}$ is a $d-$dimensional vector for every $j=1,$dots,$K.$

The likelihood of a point $x$ in a GMM is given as

$p(x|)={\displaystyle \underset{j=1}{\overset{K}{}}}{p}_{j}N(x,{}^{(i)},{}_j^2).$

The generative model can be thought of first selecting the component jin$\{1,$dots,$K\},$ which is modeled

using the multinomial distribution with parameters $p_1,$dots,$p_K,$ and then selecting a point $x$ from the Gaussian

component $N({}^{(j)},{}_j^2).$