SVD and Applications - Olga Sorkine-Hornung

SVD and Applications

Olga Sorkine-Hornung

December 23, 2016

Dimensionality reduction

December 23, 2016

n

= m

σ1 σ2

σn

UA VT Σ m⇥ n m⇥ n n⇥ n n⇥ n

data vectors aj (reduced) SVD of A

Olga Sorkine-Hornung # 2


December 23, 2016

n

m

σ1 σ2

σn

UA VT Σ m⇥ n m⇥ n n⇥ n n⇥ n

data vectors aj (reduced) SVD of A

nullify the last (small) singular values!

=



December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n

≈


≈


December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n

basis for the reduced subspace


≈


December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n


coordinates of the projections

of the data vectors onto the

reduced subspace


≈

Also called PCA, Principal Component Analysis

December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n


coordinates of the projections

of the data vectors onto the

reduced subspace


The space of human faces

December 23, 2016

x1 x2 x3

xm

……

..

Concatenate grayscale pixel values into data point a 2 Rm


Database of faces ●  Many 100x100 pixel face images ●  Different people ●  Different poses ●  Different illumination ●  … ●  Each image is labeled with the

person’s name/identification

●  Hypothesis: Faces live in a low-dimensional subspace of

December 23, 2016

R10,000


PCA on face database: Eigenfaces

December 23, 2016

average face

aavg =1

n

nX

j=1

aj



December 23, 2016

average face

aavg =1

n

nX

j=1

aj

data vectors aj := aj � aavg



December 23, 2016

average face

n

m

Am⇥ n

aavg =1

n

nX

j=1

aj




December 23, 2016

average face

n

≈ m

σ1 σk

σn

Uk A VkT Σk

m⇥ n m⇥ k k ⇥ k k ⇥ naavg =

1

n

nX

j=1

aj




December 23, 2016

average face

aavg =1

n

nX

j=1

aj

u1 u2 u3 u4

u5 u6 u7 u8

k=8 first principal components (columns of U)



December 23, 2016

u1 u2 u3 u4

u5 u6 u7 u8

k=8 first principal components (columns of U)

Coordinates of each projection of each aj onto the subspace of eigenfaces:

VkT Σk

k ⇥ n

Each face is now represented by k numbers!


Face recognition

December 23, 2016

downsample to 100x100 = m pixels,

subtract aavg aproject onto subspace,

i.e. calculate coordinates w.r.t. uj

⇠1 = aTu1

⇠2 = aTu2

⇠8 = aTu8

...

m⇥ 1


input image

Face recognition

December 23, 2016

≈ + ⇠2 + ⇠3 + ⇠4

+ ⇠5 + ⇠6 + ⇠7 + ⇠8

+ ⇠1

u1 u2 u3 u4

u5 u6 u7 u8

a aavg u1

Compare ⇠ = (⇠1, . . . , ⇠k)Twith the coordinate vectors ⇠j of

˜aj in the database

Find the “nearest neighbor” j⇤ such that k⇠ � ⇠j⇤k is the smallest


Face recognition

December 23, 2016

≈ + ⇠2 + ⇠3 + ⇠4

+ ⇠5 + ⇠6 + ⇠7 + ⇠8

+ ⇠1

u1 u2 u3 u4

u5 u6 u7 u8

a aavg u1

Compare ⇠ = (⇠1, . . . , ⇠k)Twith the coordinate vectors ⇠j of

˜aj in the database

Find the “nearest neighbor” j⇤ such that k⇠ � ⇠j⇤k is the smallest

Olga Sorkine-Hornung # 19 most likely this face in the database is labeled as “Marc Pollefeys” J

Face recognition ●  Searching for the nearest neighbor in a k-dimensional space is very

fast if k is small! And gives quite accurate results

●  Searching for the most similar face in a m=10,000-dimensional space is practically infeasible

●  Most 10,000-dimensional vectors do not represent a human face!

●  This kind of dimensionality reduction is the basis of many various machine learning methods.

December 23, 2016 Olga Sorkine-Hornung # 20

SVD and Applications - Olga Sorkine-Hornung

Documents

Transcript of SVD and Applications - Olga Sorkine-Hornung