Multidimensional Scaling

srsr

srsr

srsr

srsr

E

,

,

||

|

2

2

2

2

xx

xxxgxg

xx

xxzz

X

2

Given pairwise distances between N points,

dij, i,j =1,...,N

place on a low-dim map s.t. distances are preserved.

z = g (x | θ ) Find θ that min Sammon stress(Sammon Mapping gradient descent)

Map of Europe by MDS

3

Map from CIA – The World Factbook: http://www.cia.gov/

PCA, Sammon mapping (Welfare and poverty of one country)

4

Isomap A manifold of dimension n is a

space that near each point resembles n-dimensional Euclidean space.

Geodesic distance is the distance along the manifoldthat the data lies in, as opposed to the Euclidean distance in the input space

5

Isomap Instances r and s are connected in the graph if

||xr-xs||<e or if xs is one of the k neighbors of xr

The edge length is ||xr-xs||

For two nodes r and s not connected, the distance is equal to the shortest path between them

Once the NxN distance matrix is thus formed, use MDS to find a lower-dimensional mapping

6

7-150 -100 -50 0 50 100 150

-150

-100

-50

0

50

100

150Optdigits after Isomap (with neighborhood graph).

0

0

7

4

6

2

55

08

71

9 5

3

0

4

7

84

7

85

9

1

2

0

6

1

8

7

0

7

6

9

1

93

94

9

2

1

99

6

43

2

8

2

7

1

4

6

2

0

4

6

37 1

0

2

2

5

2

4

81

73

0

3 377

9

13

3

4

3

4

2

889 8

4

71

6

9

4

0

1 3

6

2

Matlab source from http://web.mit.edu/cocosci/isomap/isomap.html

Swissroll data (Isomap)

8

9

10

Locally Linear Embedding LLE has several advantages over Isomap

Faster optimization and better results with many problems

1. Given xr find its neighbors xs(r)

2. Find Wrs that minimize

3. Find the new coordinates zr that minimize (eigen value problem)

11

2

r s

srrs

rXE )()|( xWxW

2

r s

srrs

r zzE )()|( WWz

12

LLE on Optdigits

13

-3.5 -3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.51

1

1

1

1

1

00

7

4

6

2

5

5

0

8

7

1

9

5

3

0

47

84

7

8

5

91

2

0

6

18

7

0

76

91

9

3

9

4

9

2

1

99

6

432

82

7

14

6

2

0

46

3

7

1

0

22 52

48

1

7

3

0

33

77

91

334

342

88

98 4

7

1

6 94

0

1

36

2

Matlab source from http://www.cs.toronto.edu/~roweis/lle/code.html

14

15

16

T-distributed Stochastic Neighbor Embedding (t-SNE)

17

<고차원> <저차원>

yi 에대해미분 gradient descent

Limitations of MDS, Isomap, LLE MDS, Isomap, and LLE do not learn a general mapping

function that will allow mapping a new test point

The new point should be added to the dataset and the whole algorithm needs to be run once more

So, we can not use them as features for classification

18