Post on 02-Sep-2020
1
SVD; PCA
2
Singular Value Decomposition
A = UΣVT = λ1
|u1
|
(− vT1
−)+
· · ·+ λr
|ur
|
(− vTr −
)
UTU = I,VT
V = VVT = I
Gene Golub’s plate
3
Right svectors are evecs of A^T A
• For any matrix A
ATA = VΣ
TUTUΣV
T
= V(ΣTΣ)VT
(ATA)V = V(ΣTΣ) = VD
4
Left svectors are evecs of A A^T
AAT = UΣV
TVΣ
TUT
= U(ΣΣT )UT
(AAT )U = U(ΣΣT ) = UD
5
Truncated SVD
A = U:,1:kΣ1:k,1:kVT1:,1:k = λ1
|u1
|
(− vT1
−)+
· · ·+ λk
|uk
|
(− vTk −
)
Spectrum of singular values Rank k approximation to matrix
≈m
n k
k
k
A UΣ VT
n
6
SVD on images
• Run demo
load clown
[U,S,V] = svd(X,0);
ranks = [1 2 5 10 20 rank(X)];
for k=ranks(:)’
Xhat = (U(:,1:k)*S(1:k,1:k)*V(:,1:k)’);
image(Xhat);
end
7
Clown example
1 2 5
10 20 200
8
Space savings
A ≈ U:,1:kΣ1:k,1:kVT1:,1:k
m× n ≈ (m× k) (k) (n× k) = (m+ n+ 1)k
200× 320 = 64, 000 → (200 + 320 + 1)20 = 10, 420
9
Principal Components Analysis
-5 -4 -3 -2 -1 0 1 2 3 4 5
-3
-2
-1
0
1
2
3
4
xi =∑k
j=1 zijvj .
scores Loadings (basis)
10
PCA on height-weight data
-4 -3 -2 -1 0 1 2 3 4-4
-3
-2
-1
0
1
2
3
4
5
height
wei
ght
55 60 65 70 75 80 8580
100
120
140
160
180
200
220
240
260
280
height
wei
ght
11
Applications
• Visualizing high dim data
• Reducing dim for nearest neighbor classifier
12
Visualizing data
• Project 256 dimensional vectors (representing 16x16 images of digits) into 2D
-1000 -500 0 500 1000 1500
-1000
-500
0
500
1000
1500
1
1
1
1
1
2 22 2
2
3
3
3 33
44
4
44
5
55
5
5
6
6
6
6
6
7
77
77 8
8
8
8
8
9
99
99
0
0
0
0
0
13
Embed vectors into their z1, z2 coords
20 40 60 80 100 120 140 160 180 200
20
40
60
80
100
120
140
160
180
200
14
Nearest neighbor classifier
• Look up k nearest neighbors in training set.
• Return majority vote of their labels.• For k=1, we have
D(x,x’) = Euclidean distance• Use PCA to embed in low dimensions, compute
distances there
y = y(argminiD(xi,x
∗))
15
Eigenfaces
test imagesclosest match in training set using K=4
K=4
16
Misclassification rate vs K
0 20 40 60 80 100 120 140 1600
5
10
15
20
25
30
35
40
PCA dimensionality
num
ber
of m
iscl
assi
ficat
ion
erro
rs o
n te
st s
et
17
Eigenfaces
test images closest match in training set using K=10
K=10