Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider...

32
Multivariate Normal Distribution I Nipun Batra February 12, 2020 IIT Gandhinagar

Transcript of Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider...

Page 1: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Multivariate Normal Distribution I

Nipun Batra

February 12, 2020

IIT Gandhinagar

Page 2: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution - i

The probability density of univariate Gaussian is given as:

f (x) =1

σ√

2πe−

12 ( x−µ

σ )2

also, given as

f (x) ∼ N (µ, σ2)

with mean µ ∈ R and variance σ2 > 0

1

Page 3: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

Pop Quiz: Why is the denominator the way it is? Let the

normalizing constant be c and let g(x) = e−12 ( x−µ

σ )2

.

1 =

∫ ∞−∞

c · g(x)dx

1 =

∫ ∞−∞

ce−(x−u)2

2σ2 dx

Let’s substitute x−u√2σ

with t.

1 =

∫ ∞−∞

ce−t2dt ×

√2σ

1 =√

2σc × 2

∫ ∞0

e−t2dt

2

Page 4: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

Pop Quiz: Why is the denominator the way it is? Let the

normalizing constant be c and let g(x) = e−12 ( x−µ

σ )2

.

1 =

∫ ∞−∞

c · g(x)dx

1 =

∫ ∞−∞

ce−(x−u)2

2σ2 dx

Let’s substitute x−u√2σ

with t.

1 =

∫ ∞−∞

ce−t2dt ×

√2σ

1 =√

2σc × 2

∫ ∞0

e−t2dt

2

Page 5: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

Pop Quiz: Why is the denominator the way it is? Let the

normalizing constant be c and let g(x) = e−12 ( x−µ

σ )2

.

1 =

∫ ∞−∞

c · g(x)dx

1 =

∫ ∞−∞

ce−(x−u)2

2σ2 dx

Let’s substitute x−u√2σ

with t.

1 =

∫ ∞−∞

ce−t2dt ×

√2σ

1 =√

2σc × 2

∫ ∞0

e−t2dt

2

Page 6: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

Pop Quiz: Why is the denominator the way it is? Let the

normalizing constant be c and let g(x) = e−12 ( x−µ

σ )2

.

1 =

∫ ∞−∞

c · g(x)dx

1 =

∫ ∞−∞

ce−(x−u)2

2σ2 dx

Let’s substitute x−u√2σ

with t.

1 =

∫ ∞−∞

ce−t2dt ×

√2σ

1 =√

2σc × 2

∫ ∞0

e−t2dt

2

Page 7: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

Pop Quiz: Why is the denominator the way it is? Let the

normalizing constant be c and let g(x) = e−12 ( x−µ

σ )2

.

1 =

∫ ∞−∞

c · g(x)dx

1 =

∫ ∞−∞

ce−(x−u)2

2σ2 dx

Let’s substitute x−u√2σ

with t.

1 =

∫ ∞−∞

ce−t2dt ×

√2σ

1 =√

2σc × 2

∫ ∞0

e−t2dt

2

Page 8: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

2√π

∫ ∞0

e−t2dt

The above expression is called error function and is it’s value is

denoted by erf (t). In our case, we want erf (∞) which is equal to

1.

1 =√

2πσc × 2√π

∫ ∞0

e−t2dt

1 =√

2πσc × 1

1√2πσ

= c

3

Page 9: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

2√π

∫ ∞0

e−t2dt

The above expression is called error function and is it’s value is

denoted by erf (t). In our case, we want erf (∞) which is equal to

1.

1 =√

2πσc × 2√π

∫ ∞0

e−t2dt

1 =√

2πσc × 1

1√2πσ

= c

3

Page 10: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

2√π

∫ ∞0

e−t2dt

The above expression is called error function and is it’s value is

denoted by erf (t). In our case, we want erf (∞) which is equal to

1.

1 =√

2πσc × 2√π

∫ ∞0

e−t2dt

1 =√

2πσc × 1

1√2πσ

= c

3

Page 11: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

2√π

∫ ∞0

e−t2dt

The above expression is called error function and is it’s value is

denoted by erf (t). In our case, we want erf (∞) which is equal to

1.

1 =√

2πσc × 2√π

∫ ∞0

e−t2dt

1 =√

2πσc × 1

1√2πσ

= c

3

Page 12: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

−2 0 2

X

−0.02

−0.01

0.00

0.01

0.02Sample Point

4

Page 13: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

−2 0 2 4

Value Bins

0.0

0.1

0.2

0.3

0.4N

orm

ali

zed

Fre

qu

ency

5

Page 14: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Univariate Normal Distribution

−5.0 −2.5 0.0 2.5 5.0

X

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Est

imate

dP

robabilit

y

Variance = 0.5

−5.0 −2.5 0.0 2.5 5.0

X

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Est

imate

dP

robabilit

y

Variance = 1

−5 0 5

X

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Est

imate

dP

robabilit

y

Variance = 2

6

Page 15: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Bivariate normal distribution of two-dimensional random vector

X =

[X1

X2

]

X =

(X1

X2

)∼ N2(µ,Σ)

where, mean vector µ =

[µ1

µ2

]=

[E[X1]

E[X2]

]and, covariance matrix Σ

Σi ,j := E[(Xi − µi )(Xj − µj)] = Cov[Xi ,Xj ]

7

Page 16: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Question: What is Cov(X, X)?

Answer: Var(X) = Cov(X, X) = E[(X − E[X ])]2

In the case of univariate normal, Var(X) is written as σ2

Question: What is the relation between Σi ,j and Σj ,i?

Answer: They are the same!

Question: What can we say about the covariance matrix Σ?

Answer: It is symmetric. Thus Σ = ΣT

8

Page 17: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Question: What is Cov(X, X)?

Answer: Var(X) = Cov(X, X) = E[(X − E[X ])]2

In the case of univariate normal, Var(X) is written as σ2

Question: What is the relation between Σi ,j and Σj ,i?

Answer: They are the same!

Question: What can we say about the covariance matrix Σ?

Answer: It is symmetric. Thus Σ = ΣT

8

Page 18: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Question: What is Cov(X, X)?

Answer: Var(X) = Cov(X, X) = E[(X − E[X ])]2

In the case of univariate normal, Var(X) is written as σ2

Question: What is the relation between Σi ,j and Σj ,i?

Answer: They are the same!

Question: What can we say about the covariance matrix Σ?

Answer: It is symmetric. Thus Σ = ΣT

8

Page 19: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Question: What is Cov(X, X)?

Answer: Var(X) = Cov(X, X) = E[(X − E[X ])]2

In the case of univariate normal, Var(X) is written as σ2

Question: What is the relation between Σi ,j and Σj ,i?

Answer: They are the same!

Question: What can we say about the covariance matrix Σ?

Answer: It is symmetric. Thus Σ = ΣT

8

Page 20: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Question: What is Cov(X, X)?

Answer: Var(X) = Cov(X, X) = E[(X − E[X ])]2

In the case of univariate normal, Var(X) is written as σ2

Question: What is the relation between Σi ,j and Σj ,i?

Answer: They are the same!

Question: What can we say about the covariance matrix Σ?

Answer: It is symmetric. Thus Σ = ΣT

8

Page 21: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Normal Distribution

Question: What is Cov(X, X)?

Answer: Var(X) = Cov(X, X) = E[(X − E[X ])]2

In the case of univariate normal, Var(X) is written as σ2

Question: What is the relation between Σi ,j and Σj ,i?

Answer: They are the same!

Question: What can we say about the covariance matrix Σ?

Answer: It is symmetric. Thus Σ = ΣT

8

Page 22: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Correlation and Covariance

If X and Y are two random variables, with means (expected

values) µX and µY and standard deviations σX and σY ,

respectively, then their covariance and correlation are as follows:

covXY = σXY = E [(X − µX ) (Y − µY )]

corrXY = ρXY = E [(X − µX ) (Y − µY )]/(σXσY )

so that

ρXY = σXY /(σXσY )

where E is the expected value operator.

9

Page 23: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Correlation and Covariance

If X and Y are two random variables, with means (expected

values) µX and µY and standard deviations σX and σY ,

respectively, then their covariance and correlation are as follows:

covXY = σXY = E [(X − µX ) (Y − µY )]

corrXY = ρXY = E [(X − µX ) (Y − µY )]/(σXσY )

so that

ρXY = σXY /(σXσY )

where E is the expected value operator.

9

Page 24: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Correlation and Covariance

If X and Y are two random variables, with means (expected

values) µX and µY and standard deviations σX and σY ,

respectively, then their covariance and correlation are as follows:

covXY = σXY = E [(X − µX ) (Y − µY )]

corrXY = ρXY = E [(X − µX ) (Y − µY )]/(σXσY )

so that

ρXY = σXY /(σXσY )

where E is the expected value operator.

9

Page 25: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

PDF of bivariate normal distribution

We might have seen that

fX (X1,X2) =exp(−1

2 (X − µ)TΣ−1(X − µ))

2π|Σ|12

How do we get such a weird looking formula?!

10

Page 26: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

PDF of bivariate normal with no cross-correlation

Let us assume no correlation between X1 and X2.

We have Σ =

[σ2

1 0

0 σ22

]We have fX (X1,X2) = fX (X1)fX (X2)

=1

σ1

√2π

e− 1

2

(X1−µ1

σ1

)2

× 1

σ2

√2π

e− 1

2

(X2−µ2

σ2

)2

=1

σ1σ22πe− 1

2{(

X1−µ1σ1

)2+(

X2−µ2σ2

)2}

11

Page 27: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

PDF of bivariate normal with no cross-correlation

Let us consider only the exponential part for now

Q =(X1−µ1σ1

)2+(X2−µ2σ2

)2

Question: Can you write Q in the form of vectors X and µ?

=[X1 − µ1 X2 − µ2

]1×2

g(Σ)2×2

[X1 − µ1

X2 − µ2

]2×1

Here g(Σ) is a matrix function of Σ that will result in σ21 like

terms in the denominator; also there is no cross-terms indicating

zeros in right diagonal!

g(Σ) =

[1σ2

10

0 1σ2

2

]2×2

= 1σ2

1σ22

[σ2

2 0

0 σ21

]2×2

= 1|Σ| adj(Σ) = Σ−1

12

Page 28: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

PDF of bivariate normal with no cross-correlation

Let us consider the normalizing constant part now. M = 1σ1σ22π

= 1

2π×|Σ| 12

13

Page 29: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Gaussian samples with cross-correlation = 0

−4 −3 −2 −1 0 1 2 3 4

X1

−4

−2

0

2

4

X2

14

Page 30: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Bivariate Gaussian samples with cross-correlation 6= 0

−4 −2 0 2 4

X1

−4

−2

0

2

4

X2

15

Page 31: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Intuition for Multivariate Gaussian

Let us assume no correlation between the elements of X. This

means Σ is a diagonal matrix.

We have Σ =

σ1 0. . .

0 σn

And,

p(X;µ,Σ) =1

(2π)n2 |Σ|

12

exp

(−1

2(X− µ)TΣ−1(X− µ)

)

As seen in the case for univariate Gaussians, we can write the

following for the multivariate case,

We have fX (X1, · · · ,Xn) = fX (X1)× · · · × fX (Xn)

16

Page 32: Multivariate Normal Distribution IPDF of bivariate normal with no cross-correlation Let us consider only the exponential part for now Q = X1 1 ˙1 2 + X2 2 ˙2 2 Question: Can you

Intuition for Multivariate Gaussian

Now,

=1

σ1

√2π

e− 1

2

(X1−µ1

σ1

)2

× · · · × 1

σn√

2πe− 1

2

(Xn−µn

σn

)2

=1

σ1 · · ·σn√

2πn2

e− 1

2{(

X1−µ1σ1

)2+···+

(Xn−µn

σn

)2}

Taking all√

2π together, we get√

2πn2 .

Similarly, taking all σ together, we get σ1 · · ·σn. Which can be

written as |Σ|12 , given the determinant of a digonal matrix is the

multiplication of it’s diagonal elements.

17