of 29

• date post

15-Jan-2017
• Category

Documents

• view

218

0

Embed Size (px)

Transcript of Eigenvectors and SVD

• 1

Eigenvectors and SVD

• 2

Eigenvectors of a square matrix

Definition

Intuition: x is unchanged by A (except for scaling)

Examples: axis of rotation, stationary distribution of a Markov chain

Ax = x, x = 0 .

• 3

Diagonalization

Stack up evec equation to get

Where

If evecs are linearly indep, X is invertible, so

AX = X

X Rnn =

| | |x1 x2 xn| | |

, = diag(1, . . . , n) .

A = XX1.

• 4

Evecs of symmetric matrix

All evals are real (not complex)

Evecs are orthonormal

So U is orthogonal matrix

uTi uj = 0 if i = j, uTi ui = 1

UTU = UUT = I

• 5

Diagonalizing a symmetric matrix

We haveA = UUT =

n

i=1

iuiuTi

A = =

| | |u1 u2 un| | |

12

. . .

n

uT1

uT2

... uTn

= 1

|u1|

( uT1

)+ + n

|un|

( uTn )

• 6

Transformation by an orthogonal matrix

Consider a vector x transformed by the orthogonal matrix U to give

The length of the vector is preserved since

The angle between vectors is preserved

Thus multiplication by U can be interpreted as a rigid rotation of the coordinate system.

x = Ux

||x||2 = xT x = xTUTUTx = xTx = ||x||2

xT y = xTUUy = xTy

• 7

Geometry of diagonalization

Let A be a linear transformation. We can always decompose this into a rotation U, a scaling , and a reverse rotation UT=U-1.

Hence A = U UT. The inverse mapping is given by A-1 = U -1 UT

A =

m

i=1

iuiuTi

A1 =m

i=1

1

iuiu

Ti

• 8

Matlab example

Given

Diagonalize

check

A =

1.5 0.5 00.5 1.5 00 0 3

[U,D]=eig(A)

U =

-0.7071 -0.7071 0

-0.7071 0.7071 0

0 0 1.0000

D =

1 0 0

0 2 0

0 0 3

Rot(45)

Scale(1,2,3)

>> U*D*U

ans =

1.5000 -0.5000 0

-0.5000 1.5000 0

0 0 3.0000

• 9

Positive definite matrices

A matrix A is pd if xT A x > 0 for any non-zero vector x.

Hence all the evecs of a pd matrix are positive

A matrix is positive semi definite (psd) if i >= 0. A matrix of all positive entries is not necessarily pd;

conversely, a pd matrix can have negative entries

Aui = iui

uTi Aui = iuTi ui = i > 0

> [u,v] = eig([1 2; 3 4])

u =

-0.8246 -0.4160

0.5658 -0.9094

v =

-0.3723 0

0 5.3723

[u,v]=eig([2 -1; -1 2])

u =

-0.7071 -0.7071

-0.7071 0.7071

v =

1 0

0 3

• 10

Multivariate Gaussian

Multivariate Normal (MVN)

Exponent is the Mahalanobis distance between x and

is the covariance matrix (symmetric positive definite)

N (x|,)def=

1

(2)p/2||1/2exp[ 1

2(x )T1(x )]

= (x )T1(x )

xTx > 0 x

• 11

Bivariate Gaussian

Covariance matrix is

where the correlation coefficient is

and satisfies -1 1

Density is

=

(2x xy

xy 2y

)

=Cov(X,Y )

V ar(X)V ar(Y )

p(x, y) =1

2xy1 2

exp

(

1

2(1 2)

(x2

2x+y2

2y

2xy

(xy)

))

• 12

Spherical, diagonal, full covariance

=

(2 00 2

)

=

(2x 00 2y

) =

(2x xy

xy 2y

)

• 13

Surface plots

• 14

Matlab plotting code

stepSize = 0.5;

[x,y] = meshgrid(-5:stepSize:5,-5:stepSize:5);

[r,c]=size(x);

data = [x(:) y(:)];

p = mvnpdf(data, mu, S);

p = reshape(p, r, c);

surfc(x,y,p); % 3D plot

contour(x,y,p); % Plot contours

• 15

Visualizing a covariance matrix

Let = U UT. Hence

Let y = U(x-) be a transformed coordinate system, translated by and rotated by U. Then

1 = UT1U1 = U1U =

p

i=1

1

iuiu

Ti

(x )T1(x ) = (x )T

(p

i=1

1

iuiu

Ti

)

(x )

=

p

i=1

1

i(x )Tuiu

Ti (x ) =

p

i=1

y2ii

• 16

Visualizing a covariance matrix

From the previous slide

Recall that the equation for an ellipse in 2D is

Hence the contours of equiprobability are elliptical, with axes given by the evecs and scales given by the evals of

(x )T1(x ) =

p

i=1

y2ii

y21

1+y22

2= 1

• 17

Visualizing a covariance matrix

Let X ~ N(0,I) be points on a 2d circle.

If

Then

So we can map a set of points on a circle to points on an ellipse

Y = U1

2X

1

2 = diag(ii)

Cov[y] = U1

2Cov[x]1

2UT = U1

21

2UT =

• 18

Implementation in Matlab

Y = U1

2X

1

2 = diag(ii)

function h=gaussPlot2d(mu, Sigma, color)

[U, D] = eig(Sigma);

n = 100;

t = linspace(0, 2*pi, n);

xy = [cos(t); sin(t)];

k = 6; %k = sqrt(chi2inv(0.95, 2));

w = (k * U * sqrt(D)) * xy;

z = repmat(mu, [1 n]) + w;

h = plot(z(1, :), z(2, :), color); axis(equal

• 19

Standardizing the data

We can subtract off the mean and divide by the standard deviation of each dimension to get the following (for case i=1:n and dimension j=1:d)

Then E[Y]=0 and Var[Yj]=1.

However, Cov[Y] might still be elliptical due to correlation amongst the dimensions.

yij =xij xjj

• 20

Whitening the data

Let X ~ N(,) and = U UT. To remove any correlation, we can apply the

following linear transformation

In Matlab

Y = 1

2UTX

1

2 = diag(1/ii)

[U,D] = eig(cov(X));

Y = sqrt(inv(D)) * U * X;

• 21

Whitening: example

• 22

Whitening: proof

Let

Using

we have

and

Y = 1

2UTX

1

2 = diag(1/ii)

Cov[AX] = ACov[X ]AT

Cov[Y ] = 1

2UTU1

2

= 1

2UT (UUT )U1

2

= 1

21

2 = I

EY = 1

2UTE[X ]

• 23

Singular Value Decomposition

A = UVT = 1

|u1|

( vT1

)+

+ r

|ur|

( vTr )

• 24

Right svectors are evecs of A^T A

For any matrix A

ATA = VTUT UVT

= V(T)VT

(ATA)V = V(T) = VD

• 25

Left svectors are evecs of A A^T

AAT = UVT VTUT

= U(T )UT

(AAT )U = U(T ) = UD

• 26

Truncated SVD

A = U:,1:k1:k,1:kVT1:,1:k = 1

|u1|

( vT1

)+

+ k

|uk|

( vTk )

Spectrum of singular values

Rank k approximation to matrix

• 27

SVD on images

Run demo

[U,S,V] = svd(X,0);

ranks = [1 2 5 10 20 rank(X)];

for k=ranks(:)

Xhat = (U(:,1:k)*S(1:k,1:k)*V(:,1:k));

image(Xhat);

end

• 28

Clown example

1 2 5

10 20 200

• 29

Space savings

A U:,1:k1:k,1:kVT1:,1:k

m n (m k) (k) (n k) = (m+ n+ 1)k

200 320 = 64, 000 (200 + 320 + 1)20 = 10, 420