Gaussian Mixture Model (GMM) using Expectationvplab/courses/DVP/PDF/gmm.pdf · ML Method for...
15
Gaussian Mixture Model (GMM) using Expectation Maximization (EM) Technique Book : C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006
Transcript of Gaussian Mixture Model (GMM) using Expectationvplab/courses/DVP/PDF/gmm.pdf · ML Method for...
Gaussian Mixture Model (GMM) using Expectation Maximization
(EM) Technique
Book : C.M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006
The Gaussian Distribution
Univariate Gaussian Distribution
2
2
σ2μ)(x
2e
πσ21σμ,|x
−−=)Ν(
Multi-Variate Gaussian Distribution
( ) ( ) ( )
−−−
= − μxμx
21exp
2π1μ,x 1T
21/)|Ν(
mean covariance
We need to estimate these parameters of a distribution One method – Maximum Likelihood (ML) Estimation.
mean variance
ML Method for estimating parameters Consider log of Gaussian Distribution
( ) ( )μxμx21ln
21)2ln(
21)μ,|x(pln 1T −−−−−= −π
Take the derivative and equate it to zero
0),|x(pln =∂
∂μμ
0),|x(pln =∂
∂ μ
=
=N
1nnML x
N1μ ( )( )T
MLn
N
1nMLnML xx
N1 μμ −−=
=
Where, N is the number of samples or data points
Gaussian Mixtures Linear super-position of Gaussians
),|x()x(p kk
K
1kk =
=
μπ N
Number of Gaussians Mixing coefficient: weightage for each Gaussian dist.
Normalization and positivity require ,10 k ≤≤ π 1K
1kk =
=
π
ML does not work here as there is no closed form solutionParameters can be calculated using Expectation Maximization (EM) technique
Consider log likelihood
= ==
==N
1n
K
1kkknk
N
1nn ),|x(Nln)x(pln),,|X(pln μππμ
Example: Mixture of 3 Gaussian
0 0.5 10
0.5
1
(a)0 0.5 1
0
0.5
1
(b)
Latent variable: posterior prob. We can think of the mixing coefficients as prior
probabilities for the components For a given value of ‘x’, we can evaluate the
corresponding posterior probabilities, called responsibilities
)x(p)k|x(p)k(p)x|k(p)x(k ==γ
=
= K
1jjjj
kkk
),|x(
),|x(
μπ
μπ
N
N
From Bayes rule
Latent Variable N
N where, kk =π
Interpret Nk as the effective no. of points assigned to cluster k.
Expectation Maximization EM algorithm is an iterative optimization technique
which is operated locally
Initial point
Optimal point
Estimation step: for given parameter values we can compute the expected values of the latent variable.
Maximization step: updates the parameters of our model based on the latent variable calculated using ML method.
EM Algorithm for GMMGiven a Gaussian mixture model, the goal is to maximize the likelihood function with respect to the parameters comprising the means and covariances of the components and the mixing coefficients).
1. Initialize the means , covariances and mixing coefficients , and evaluate the initial value of the log likelihood.
jμ jjπ
2. E step. Evaluate the responsibilities using the current parameter values
=
= K
1jjjj
kkkj
),|x(
),|x()x(μπ
μπγN
N
EM Algorithm for GMM3. M step. Re-estimate the parameters using the current
responsibilities
=
== N
1nnj
N
1nnnj
j
)x(
x)x(
γ
γμ )x(
N1
n
N
1njj
=
= γπ( )( )
=
=
−−= N
1nnj
N
1n
Tjnjnnj
j
)x(
xx)x(
γ
μμγ
4. Evaluate log likelihood
If there is no convergence, return to step 2.
= =
=N
1n
K
1kkknk ),|x(Nln),,|X(pln μππμ
EM Algorithm : Example
EM Algorithm : Example
EM Algorithm : Example
EM Algorithm : Example
EM Algorithm : Example
EM Algorithm : Example
![Formulario de Prec´ . · PDF file11) R tanudu = ln[secu] = −ln[cosu]+C 12) R cotudu = lnsenu 13) R secudu = ln[secu+tanu] = ln u tan au u 2 + π 4 14) R cscudu = ln[cscu−cotu]](https://static.fdocument.org/doc/165x107/5aba216c7f8b9ac1058ea568/formulario-de-r-tanudu-lnsecu-lncosuc-12-r-cotudu-lnsenu-13-r.jpg)
