SVD and Applications

Olga Sorkine-Hornung

December 23, 2016

Dimensionality reduction

December 23, 2016

n

=  m

σ1 σ2

σn

UA VT Σ m⇥ n m⇥ n n⇥ n n⇥ n

data vectors aj (reduced) SVD of A

Olga Sorkine-Hornung # 2

Dimensionality reduction

December 23, 2016

n

m

σ1 σ2

σn

UA VT Σ m⇥ n m⇥ n n⇥ n n⇥ n

data vectors aj (reduced) SVD of A

nullify the last (small) singular values!

=  

Olga Sorkine-Hornung # 3

Dimensionality reduction

December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n

≈  

Olga Sorkine-Hornung # 4

≈  

Dimensionality reduction

December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n

basis for the reduced subspace

Olga Sorkine-Hornung # 5

≈  

Dimensionality reduction

December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n

basis for the reduced subspace

coordinates of the projections

of the data vectors onto the

reduced subspace

Olga Sorkine-Hornung # 6

≈  

Also called PCA, Principal Component Analysis

December 23, 2016

n

m

σ1 σk

σn

Uk A VkT Σk

m⇥ n

data vectors aj

m⇥ k k ⇥ k k ⇥ n

basis for the reduced subspace

coordinates of the projections

of the data vectors onto the

reduced subspace

Olga Sorkine-Hornung # 7

The space of human faces

December 23, 2016

x1 x2 x3

xm

……

..

Concatenate grayscale pixel values into data point a 2 Rm

Olga Sorkine-Hornung # 8

Database of faces ●  Many 100x100 pixel face images ●  Different people ●  Different poses ●  Different illumination ●  … ●  Each image is labeled with the

person’s name/identification

●  Hypothesis: Faces live in a low-dimensional subspace of

December 23, 2016

R10,000

Olga Sorkine-Hornung # 9

PCA on face database: Eigenfaces

December 23, 2016

average face

aavg =1

n

nX

j=1

aj

Olga Sorkine-Hornung # 10

PCA on face database: Eigenfaces

December 23, 2016

average face

aavg =1

n

nX

j=1

aj

data vectors aj := aj � aavg

Olga Sorkine-Hornung # 11

PCA on face database: Eigenfaces

December 23, 2016

average face

n

m

Am⇥ n

aavg =1

n

nX

j=1

aj

data vectors aj := aj � aavg

Olga Sorkine-Hornung # 12

PCA on face database: Eigenfaces

December 23, 2016

average face

n

≈  m

σ1 σk

σn

Uk A VkT Σk

m⇥ n m⇥ k k ⇥ k k ⇥ naavg =

1

n

nX

j=1

aj

data vectors aj := aj � aavg

Olga Sorkine-Hornung # 13

PCA on face database: Eigenfaces

December 23, 2016

average face

n

≈  m

σ1 σk

σn

Uk A VkT Σk

m⇥ n m⇥ k k ⇥ k k ⇥ naavg =

1

n

nX

j=1

aj

data vectors aj := aj � aavg

Olga Sorkine-Hornung # 14

PCA on face database: Eigenfaces

December 23, 2016

average face

aavg =1

n

nX

j=1

aj

u1 u2 u3 u4

u5 u6 u7 u8

k=8 first principal components (columns of U)

Olga Sorkine-Hornung # 15

PCA on face database: Eigenfaces

December 23, 2016

u1 u2 u3 u4

u5 u6 u7 u8

k=8 first principal components (columns of U)

Coordinates of each projection of each aj onto the subspace of eigenfaces:

VkT Σk

k ⇥ n

Each face is now represented by k numbers!

Olga Sorkine-Hornung # 16

Face recognition

December 23, 2016

downsample to 100x100 = m pixels,

subtract aavg aproject onto subspace,

i.e. calculate coordinates w.r.t. uj

⇠1 = aTu1

⇠2 = aTu2

⇠8 = aTu8

...

m⇥ 1

Olga Sorkine-Hornung # 17

input image

Face recognition

December 23, 2016

≈   + ⇠2 + ⇠3 + ⇠4

+ ⇠5 + ⇠6 + ⇠7 + ⇠8

+ ⇠1

u1 u2 u3 u4

u5 u6 u7 u8

a aavg u1

Compare ⇠ = (⇠1, . . . , ⇠k)Twith the coordinate vectors ⇠j of

˜aj in the database

Find the “nearest neighbor” j⇤ such that k⇠ � ⇠j⇤k is the smallest

Olga Sorkine-Hornung # 18

Face recognition

December 23, 2016

≈   + ⇠2 + ⇠3 + ⇠4

+ ⇠5 + ⇠6 + ⇠7 + ⇠8

+ ⇠1

u1 u2 u3 u4

u5 u6 u7 u8

a aavg u1

Compare ⇠ = (⇠1, . . . , ⇠k)Twith the coordinate vectors ⇠j of

˜aj in the database

Find the “nearest neighbor” j⇤ such that k⇠ � ⇠j⇤k is the smallest

Olga Sorkine-Hornung # 19 most likely this face in the database is labeled as “Marc Pollefeys” J

Face recognition ●  Searching for the nearest neighbor in a k-dimensional space is very

fast if k is small! And gives quite accurate results

●  Searching for the most similar face in a m=10,000-dimensional space is practically infeasible

●  Most 10,000-dimensional vectors do not represent a human face!

●  This kind of dimensionality reduction is the basis of many various machine learning methods.

December 23, 2016 Olga Sorkine-Hornung # 20