Eigenvectors and SVD

download Eigenvectors and SVD

of 29

  • date post

    15-Jan-2017
  • Category

    Documents

  • view

    217
  • download

    0

Embed Size (px)

Transcript of Eigenvectors and SVD

  • 1

    Eigenvectors and SVD

  • 2

    Eigenvectors of a square matrix

    Definition

    Intuition: x is unchanged by A (except for scaling)

    Examples: axis of rotation, stationary distribution of a Markov chain

    Ax = x, x = 0 .

  • 3

    Diagonalization

    Stack up evec equation to get

    Where

    If evecs are linearly indep, X is invertible, so

    AX = X

    X Rnn =

    | | |x1 x2 xn| | |

    , = diag(1, . . . , n) .

    A = XX1.

  • 4

    Evecs of symmetric matrix

    All evals are real (not complex)

    Evecs are orthonormal

    So U is orthogonal matrix

    uTi uj = 0 if i = j, uTi ui = 1

    UTU = UUT = I

  • 5

    Diagonalizing a symmetric matrix

    We haveA = UUT =

    n

    i=1

    iuiuTi

    A = =

    | | |u1 u2 un| | |

    12

    . . .

    n

    uT1

    uT2

    ... uTn

    = 1

    |u1|

    ( uT1

    )+ + n

    |un|

    ( uTn )

  • 6

    Transformation by an orthogonal matrix

    Consider a vector x transformed by the orthogonal matrix U to give

    The length of the vector is preserved since

    The angle between vectors is preserved

    Thus multiplication by U can be interpreted as a rigid rotation of the coordinate system.

    x = Ux

    ||x||2 = xT x = xTUTUTx = xTx = ||x||2

    xT y = xTUUy = xTy

  • 7

    Geometry of diagonalization

    Let A be a linear transformation. We can always decompose this into a rotation U, a scaling , and a reverse rotation UT=U-1.

    Hence A = U UT. The inverse mapping is given by A-1 = U -1 UT

    A =

    m

    i=1

    iuiuTi

    A1 =m

    i=1

    1

    iuiu

    Ti

  • 8

    Matlab example

    Given

    Diagonalize

    check

    A =

    1.5 0.5 00.5 1.5 00 0 3

    [U,D]=eig(A)

    U =

    -0.7071 -0.7071 0

    -0.7071 0.7071 0

    0 0 1.0000

    D =

    1 0 0

    0 2 0

    0 0 3

    Rot(45)

    Scale(1,2,3)

    >> U*D*U

    ans =

    1.5000 -0.5000 0

    -0.5000 1.5000 0

    0 0 3.0000

  • 9

    Positive definite matrices

    A matrix A is pd if xT A x > 0 for any non-zero vector x.

    Hence all the evecs of a pd matrix are positive

    A matrix is positive semi definite (psd) if i >= 0. A matrix of all positive entries is not necessarily pd;

    conversely, a pd matrix can have negative entries

    Aui = iui

    uTi Aui = iuTi ui = i > 0

    > [u,v] = eig([1 2; 3 4])

    u =

    -0.8246 -0.4160

    0.5658 -0.9094

    v =

    -0.3723 0

    0 5.3723

    [u,v]=eig([2 -1; -1 2])

    u =

    -0.7071 -0.7071

    -0.7071 0.7071

    v =

    1 0

    0 3

  • 10

    Multivariate Gaussian

    Multivariate Normal (MVN)

    Exponent is the Mahalanobis distance between x and

    is the covariance matrix (symmetric positive definite)

    N (x|,)def=

    1

    (2)p/2||1/2exp[ 1

    2(x )T1(x )]

    = (x )T1(x )

    xTx > 0 x

  • 11

    Bivariate Gaussian

    Covariance matrix is

    where the correlation coefficient is

    and satisfies -1 1

    Density is

    =

    (2x xy

    xy 2y

    )

    =Cov(X,Y )

    V ar(X)V ar(Y )

    p(x, y) =1

    2xy1 2

    exp

    (

    1

    2(1 2)

    (x2

    2x+y2

    2y

    2xy

    (xy)

    ))

  • 12

    Spherical, diagonal, full covariance

    =

    (2 00 2

    )

    =

    (2x 00 2y

    ) =

    (2x xy

    xy 2y

    )

  • 13

    Surface plots

  • 14

    Matlab plotting code

    stepSize = 0.5;

    [x,y] = meshgrid(-5:stepSize:5,-5:stepSize:5);

    [r,c]=size(x);

    data = [x(:) y(:)];

    p = mvnpdf(data, mu, S);

    p = reshape(p, r, c);

    surfc(x,y,p); % 3D plot

    contour(x,y,p); % Plot contours

  • 15

    Visualizing a covariance matrix

    Let = U UT. Hence

    Let y = U(x-) be a transformed coordinate system, translated by and rotated by U. Then

    1 = UT1U1 = U1U =

    p

    i=1

    1

    iuiu

    Ti

    (x )T1(x ) = (x )T

    (p

    i=1

    1

    iuiu

    Ti

    )

    (x )

    =

    p

    i=1

    1

    i(x )Tuiu

    Ti (x ) =

    p

    i=1

    y2ii

  • 16

    Visualizing a covariance matrix

    From the previous slide

    Recall that the equation for an ellipse in 2D is

    Hence the contours of equiprobability are elliptical, with axes given by the evecs and scales given by the evals of

    (x )T1(x ) =

    p

    i=1

    y2ii

    y21

    1+y22

    2= 1

  • 17

    Visualizing a covariance matrix

    Let X ~ N(0,I) be points on a 2d circle.

    If

    Then

    So we can map a set of points on a circle to points on an ellipse

    Y = U1

    2X

    1

    2 = diag(ii)

    Cov[y] = U1

    2Cov[x]1

    2UT = U1

    21

    2UT =

  • 18

    Implementation in Matlab

    Y = U1

    2X

    1

    2 = diag(ii)

    function h=gaussPlot2d(mu, Sigma, color)

    [U, D] = eig(Sigma);

    n = 100;

    t = linspace(0, 2*pi, n);

    xy = [cos(t); sin(t)];

    k = 6; %k = sqrt(chi2inv(0.95, 2));

    w = (k * U * sqrt(D)) * xy;

    z = repmat(mu, [1 n]) + w;

    h = plot(z(1, :), z(2, :), color); axis(equal

  • 19

    Standardizing the data

    We can subtract off the mean and divide by the standard deviation of each dimension to get the following (for case i=1:n and dimension j=1:d)

    Then E[Y]=0 and Var[Yj]=1.

    However, Cov[Y] might still be elliptical due to correlation amongst the dimensions.

    yij =xij xjj

  • 20

    Whitening the data

    Let X ~ N(,) and = U UT. To remove any correlation, we can apply the

    following linear transformation

    In Matlab

    Y = 1

    2UTX

    1

    2 = diag(1/ii)

    [U,D] = eig(cov(X));

    Y = sqrt(inv(D)) * U * X;

  • 21

    Whitening: example

  • 22

    Whitening: proof

    Let

    Using

    we have

    and

    Y = 1

    2UTX

    1

    2 = diag(1/ii)

    Cov[AX] = ACov[X ]AT

    Cov[Y ] = 1

    2UTU1

    2

    = 1

    2UT (UUT )U1

    2

    = 1

    21

    2 = I

    EY = 1

    2UTE[X ]

  • 23

    Singular Value Decomposition

    A = UVT = 1

    |u1|

    ( vT1

    )+

    + r

    |ur|

    ( vTr )

  • 24

    Right svectors are evecs of A^T A

    For any matrix A

    ATA = VTUT UVT

    = V(T)VT

    (ATA)V = V(T) = VD

  • 25

    Left svectors are evecs of A A^T

    AAT = UVT VTUT

    = U(T )UT

    (AAT )U = U(T ) = UD

  • 26

    Truncated SVD

    A = U:,1:k1:k,1:kVT1:,1:k = 1

    |u1|

    ( vT1

    )+

    + k

    |uk|

    ( vTk )

    Spectrum of singular values

    Rank k approximation to matrix

  • 27

    SVD on images

    Run demo

    load clown

    [U,S,V] = svd(X,0);

    ranks = [1 2 5 10 20 rank(X)];

    for k=ranks(:)

    Xhat = (U(:,1:k)*S(1:k,1:k)*V(:,1:k));

    image(Xhat);

    end

  • 28

    Clown example

    1 2 5

    10 20 200

  • 29

    Space savings

    A U:,1:k1:k,1:kVT1:,1:k

    m n (m k) (k) (n k) = (m+ n+ 1)k

    200 320 = 64, 000 (200 + 320 + 1)20 = 10, 420