1

SVD; PCA

2

Singular Value Decomposition

A = UΣVT = λ1

|u1

|

(− vT1

−)+

· · ·+ λr

|ur

|

(− vTr −

)

UTU = I,VT

V = VVT = I

Gene Golub’s plate

3

Right svectors are evecs of A^T A

• For any matrix A

ATA = VΣ

TUTUΣV

T

= V(ΣTΣ)VT

(ATA)V = V(ΣTΣ) = VD

4

Left svectors are evecs of A A^T

AAT = UΣV

TVΣ

TUT

= U(ΣΣT )UT

(AAT )U = U(ΣΣT ) = UD

5

Truncated SVD

A = U:,1:kΣ1:k,1:kVT1:,1:k = λ1

|u1

|

(− vT1

−)+

· · ·+ λk

|uk

|

(− vTk −

)

Spectrum of singular values Rank k approximation to matrix

≈m

n k

k

k

A UΣ VT

n

6

SVD on images

• Run demo

load clown

[U,S,V] = svd(X,0);

ranks = [1 2 5 10 20 rank(X)];

for k=ranks(:)’

Xhat = (U(:,1:k)*S(1:k,1:k)*V(:,1:k)’);

image(Xhat);

end

7

Clown example

1 2 5

10 20 200

8

Space savings

A ≈ U:,1:kΣ1:k,1:kVT1:,1:k

m× n ≈ (m× k) (k) (n× k) = (m+ n+ 1)k

200× 320 = 64, 000 → (200 + 320 + 1)20 = 10, 420

9

Principal Components Analysis

-5 -4 -3 -2 -1 0 1 2 3 4 5

-3

-2

-1

0

1

2

3

4

xi =∑k

j=1 zijvj .

scores Loadings (basis)

10

PCA on height-weight data

-4 -3 -2 -1 0 1 2 3 4-4

-3

-2

-1

0

1

2

3

4

5

height

wei

ght

55 60 65 70 75 80 8580

100

120

140

160

180

200

220

240

260

280

height

wei

ght

11

Applications

• Visualizing high dim data

• Reducing dim for nearest neighbor classifier

12

Visualizing data

• Project 256 dimensional vectors (representing 16x16 images of digits) into 2D

-1000 -500 0 500 1000 1500

-1000

-500

0

500

1000

1500

1

1

1

1

1

2 22 2

2

3

3

3 33

44

4

44

5

55

5

5

6

6

6

6

6

7

77

77 8

8

8

8

8

9

99

99

0

0

0

0

0

13

Embed vectors into their z1, z2 coords

20 40 60 80 100 120 140 160 180 200

20

40

60

80

100

120

140

160

180

200

14

Nearest neighbor classifier

• Look up k nearest neighbors in training set.

• Return majority vote of their labels.• For k=1, we have

D(x,x’) = Euclidean distance• Use PCA to embed in low dimensions, compute

distances there

y = y(argminiD(xi,x

∗))

15

Eigenfaces

test imagesclosest match in training set using K=4

K=4

16

Misclassification rate vs K

0 20 40 60 80 100 120 140 1600

5

10

15

20

25

30

35

40

PCA dimensionality

num

ber

of m

iscl

assi

ficat

ion

erro

rs o

n te

st s

et

17

Eigenfaces

test images closest match in training set using K=10

K=10