L7 GMM EM Exercises Solutions PDF

Preview

Summary

Human-centered Assistive MACHINE LEARNING Robotics IN ROBOTICS Technische Universit ̈at M ̈unchen Exercises GMM/EM Prof. Dr.-Ing. Dongheui LeeExercise 1Consider a Gaussian mixture model in which the marginal distributionp(z)for the latent variable is givenbyp(z) =∏Kk=πzkk (where zk ∈ { 0 , 1 }and∑kz...

Description

Human-centered Assistive Robotics Technische Universit¨at M¨unchen

MACHINE LEARNING IN ROBOTICS Exercises GMM/EM

Prof. Dr.-Ing. Dongheui Lee

Exercise 1 Consider a Gaussian mixture model in which the marginal distribution p(z) for the latent variable is given K P Q by p(z) = zk = 1), and the conditional distribution p(x|z) for the π kzk (where zk ∈ {0, 1} and k=1

k

observed variable is given by p(x|z) =

K Q

k=1

N (x|µk , Σk )zk . Show that the marginal distribution p(x),

obtained by summing p(z)p(x|z) over all possible values of z, is a Gaussian mixture of the form p(x) = K P π k N (x|µk , Σk ). k=1

Solution:

p(x) can be rewritten as the marginalized joint distribution p(x, z) over all possible values of z : P p(x) = p(x, z) z

It applies for random variables that p(x, z) = p(x|z )p(z).

Hence, we obtain P p(x) = (p(x|z )p(z)) . z

The conditional distribution p(x|z) for the observed variable is given by p(x|z) =

K Q

k=1

p(z) is given as

K Q

k=1

πkzk.

N (x|µk , Σk )zk and

Inserting this into the equation results in  K K P Q zk Q zk (N (x|µk , Σk )) · p(x) = πk k=1

z

p(x) =

P z



K Q

k=1

k=1

(πkzkN (x|µk , Σk ))zk



=

K PQ

(π k N (x|µk , Σk ))zk .

z k=1

Since z can only take 1 of K possible state, we can introduce an indicator variable I such that Ikj = 1 if k = j and 0 otherwise. Now we can rewrite p(x) as p(x) =

K Q K P

j=1 k=1

(π k N (x|µk , Σk ))Ikj =

K P

j=1

π j N (x|µj , Σj )

Exercise 2 There is a one-dimensional data-set given as x1 = 1; x2 = 3; x3 = 4. The goal is to fit a Gaussian Mixture Model with two components (C1 , C2 ) onto the data. The initial model parameters are given by the means µ1 = 1; µ2 = 2, variances σ 12 = 0.1; σ22 = 0.1 and priors π 1 = 0.5; π 2 = 0.5. a) Draw the points on the horizontal axis and sketch the mean and probability density function of the initial Gaussians (vertical axis is the probability density) in the provided figure for the initial state. b) Compute the EM-Algorithm for Gaussian Mixture Models for one iteration and report priors, means and variances. c) Sketch the Gaussian components after the first iteration into the provided figure.

Initial 1.5 1 0.5 0 0

1

2

3

4

5

6

7

5

6

7

Iteration 1 1.5 1 0.5 0 0

1

2

3

4

Figure 1: Space for your solution Solution 2

a) Find the sketch in the provided figure. You can roughly sketch the form of the probability density function by computing its value at the center. Remember that the area under the PDF equals 1. Compute the PDF value for x = µk for each Gaussian k with N (x|µk , σk2) =

σk

1 √

1 exp − 2 2π



x − µk σk

2 !

(1)

N (x = µ1 |µ1 , σ 21 ) = 1.2616

N (x = µ2 |µ2 , σ 22 ) = 1.2616

To get a better impression how the Gaussian looks like, you can use the rule of thumb that at µ ± 2σ roughly 95% of the probability mass is covered. Therefore, compute this x-value to estimate where the curve approximates the horizontal axis, e.g. for C1 : √ µ1 + 2σ1 = 1 + 2 · 0.1 = 1.6325 b) Apply the EM-Algorithm for Gaussian Mixture Models. Variables with a hat symbol such as πˆk are taken from the previous computation (or in the first iteration from the initial value).

E-Step: For each point i and component k, compute the responsibility

Exemplarily for i = 1 and k = 1: γˆ1,1 =

π ˆ k N (xi |ˆ µk , σ ˆk2) γˆik = PK . ˆ j N (xi |ˆ µj , σ ˆj2) j=1 π

0.5 · N (1|1, 0.1) 0.6308 = 0.9933. = 0.6308 + 0.0043 0.5 · N (1|1, 0.1) + 0.5 · N (1|2, 0.1)

with 1

1 N (1|1, 0.1) = √ exp − 2 2π · 0.1



1−1 √ 0.1

2 !

For component C1 : 0.6308 γ1,1 = 0.6308+0.0043 = 0.9933 γ2,1 =

1.3001·10−9 1.3001·10−9 +0.0043

= 3.0590 · 10−7

·10 γ3,1 1.8056·1.8056 = 1.3888 · 10−11 10−20 +1.3001·10−9 −20

For component C2 : 0.0043 γ1,2 = 0.0043+0.6308 = 0.0067 0.0043 = 1.000 γ2,2 0.0043+1 .3001·10−9 1.3001·10 γ3,2 1.3001·10 −9 +1.8056·10−20 = 1.000 −9

M-Step: The number of points is given as N . Use the values from the E-Step to compute new priors, means and variances in the following order:

π ˆk =

N X γˆik

N

i=1

PN

µˆk = Pi=1 N

γˆik xi

i=1

σ ˆk2 = The values are given as follows:

π ˆ µˆ σ ˆ2

PN

γˆik

γˆik (xi − µˆk )2 PN ˆik i=1 γ

i=1

C1 0.3311 1.000 1.232 · 10−6

C2 0.6689 3.4917 0.2699

Exemplarily, for k = 1: π ˆ1 =

N X γˆi,1 i=1

1 = (0.9933 + 3.0590 · 10−7 + 1.3888 · 10−11 ) · 3 = 0.3311 3

PN 0.9933 · 1 + 3.0590 · 10−7 · 3 + 1.3888 · 10−11 · 4 γˆi,1 xi = = 1.000 µˆ1 = Pi=1 N 0.9933 + 3.0590 · 10−7 + 1.3888 · 10−11 ˆi,1 i=1 γ PN γˆi,1 (xi − µˆ1 )2 2 = σ ˆ1 = i=1PN γ ˆ i,1 i=1

0.9933(1 − 1.000)2 + 3.0590 · 10−7 · (3 − 1.000)2 + 1.3888 · 10−11 (4 − 1.000)2 = 1.232 · 10−6 −7 −11 0.9933 + 3.0590 · 10 + 1.3888 · 10 c) Sketch the Gaussian components after the first iteration into the provided figure by using the means and variances from subtask b). Follow the strategy given in solution for subtask a).

Initial 1.5 1 0.5 0 0

1

2

3

4

5

6

7

5

6

7

Iteration 1 1.5 1 0.5 0 0

1

2

3

4

Figure 2: Results of the EM-Algorithm. Due to the small variance of the left Gaussian, its curve is higher than the figure axis shows....