1

Problem: Consider a two class task with ω1, ω2

02211

0

...

0)(

wxwxwxw

wxwxg

ll

T

2121

0201

21

0

0

:hyperplanedecision on the Assume

x,x)xx(w

wxwwxw

x,x

T

TT

LINEAR CLASSIFIERS

2

Hence:

hyperplane on the w

0)( 0 wxwxg T

22

21

22

21

0 ,ww

)x(gz

ww

wd

3

The Perceptron Algorithm

Assume linearly separable classes, i.e.

The casefalls under the above formulation, since

•

•

2T

1T

x0x*w

x0x*w:*w

*T* wxw 0

1

0

x'x,

w

*w'w *

00 'x'wwx*w T*T

4

Our goal: Compute a solution, i.e., a hyperplane w,so that

• The steps– Define a cost function to be minimized– Choose an algorithm to minimize the cost

function– The minimum corresponds to a solution

2

1 0)(

xxwT

5

The Cost Function

• Where Y is the subset of the vectors wrongly classified by w. When Y=(empty set) a solution is achieved and

•

•

•

Yx

Tx xwwJ )()(

0)( wJ

2

1

and if 1

and if 1

xYx

xYx

x

x

0)( wJ

6

• J(w) is piecewise linear (WHY?)

The Algorithm• The philosophy of the gradient descent

is adopted.

7

• Wherever valid

•

This is the celebrated Perceptron Algorithm

old)()(

(old)(new)

www

wJw

www

xxwww

wJ

Yx Yxx

Tx

)(

)(

xtwtwYx

xt

)()1(

w

8

An example

The perceptron algorithm converges in a finite number of iteration steps to a solution if

)1(x)t(w

x)t(w)1t(w

xxt

t

t

ct

t

kk

t

t

kk

t

:e.g.

limlim0

2

0

9

A useful variant of the perceptron algorithm

It is a reward and punishment type of algorithm

otherwise )()1(

0)( ,)()1(

0)( , )()1(

2)(

)()(

1)(

)()(

twtw

x

xtwxtwtw

x

xtwxtwtw

t

tT

t

t

tT

t

10

The perceptron

shold thre

weightssynapticor synapses

0w

s'wi

It is a learning machine that learns from the training vectors via the perceptron algorithm

The network is called perceptron or neuron

11

Example: At some stage t the perceptron algorithm results in

The corresponding hyperplane is

5.0w,1w,1w 021

05.021 xx

5.0

51.0

42.1

1

75.0

2.0

)1(7.0

1

05.0

4.0

)1(7.0

5.0

1

1

)1(tw

ρ=0.7

12

Least Squares Methods If classes are linearly separable, the perceptron

output results in

If classes are NOT linearly separable, we shall compute the weights

so that the difference between• The actual output of the classifier, , and

• The desired outputs, e.g.

to be SMALL

1

021 w,...,w,w

xwT

2

1

if 1

if 1

x

x

13

SMALL, in the mean square error sense, means to choose so that the cost function

•

•

•

w

responses desired ingcorrespond the

)(minargˆ

minimum is ])[()( 2

y

wJw

xwyEwJ

w

T

14

Minimizing

where Rx the autocorrelation matrix

and the crosscorrelation vector

][ˆ

][][

)]([2

0])[()(

:in results to w.r.)(

1

2

yxERw

yxEwxxE

wxyxE

xwyEww

wJ

wwJ

x

T

T

T

][]...[][

...................................

][]...[][

][

21

12111

llll

lT

x

xxExxExxE

xxExxExxE

xxER

][

...

][

][1

yxE

yxE

yxE

l

15

SMALL in the sum error squares sense means

that is, the input xi and its

corresponding class label (±1).

),(

)()(

ii

2N

1ii

Ti

xy

xwywJ

: training pairs

i

N

ii

Ti

N

ii

N

ii

Ti

yxwxx

xwyww

wJ

11

1

2

)(

0)()(

iy

16

Pseudoinverse Matrix Define

responses desired ingcorrespond y

matrix) (an

N

1

TN

T2

T1

y

...

y

Nxl

x

...

x

x

X

matrix) (an lxN]x,...,x,x[X N21T

N

i

Tii

T xxXX1

i

N

ii

T yxyX

1

17

Thus

Assume N=l X square and invertible. Then

yX

yXXXw

yXwXX

yxwxx

TT

TT

N

i

N

iiii

Ti

1

1 1

)(ˆ

ˆ)(

)(ˆ)(

TT XXXX 1)( Pseudoinverse of X

1

111)(

XX

XXXXXXX TTTT

18

Assume N>l. Then, in general, there is no solution to satisfy all equations simultaneously:

The “solution” corresponds to the minimum sum of squares solution

unknowns l equations N ....

NTN

2T2

1T1

ywx

ywx

ywx

:ywX

yXw

19

Example:

5.0

7.0,

6.0

8.0,

4.0

7.0,

2.0

6.0,

6.0

4.0:

3.0

3.0,

7.0

2.0,

4.0

1.0,

5.0

6.0,

5.0

4.0:

2

1

y

..

..

..

..

..

..

..

..

..

..

X

1

1

1

1

1

1

1

1

1

1

15070

16080

14070

12060

16040

13030

17020

14010

15060

15040

20

34.1

24.0

13.3

)(

0.0

1.0

6.1

,

107.48.4

7.441.224.2

8.424.28.2

1 yXXXw

yXXX

TT

TT

21

The goal: Given two linearly separable classes, design the classifier

that leaves the maximum margin from both classes

0)( 0 wxwxg T

SUPPORT VECTOR MACHINES

22

Margin: Each hyperplane is characterized by• Its direction in space, i.e.,

• Its position in space, i.e.,

• For EACH direction,choose the hyperplane that leaves the SAME distance from the nearest points from each class. The margin is twice this distance.

w

0w

w

23

The distance of a point from a hyperplane is given by

Scale, so that at the nearest points from each class the discriminant function is ±1:

Thus the margin is given by

Also, the following is valid

x̂

, , 0ww

w

)x̂(gz x̂

21 for 1)( and for 1)x( 1)( xggxg

www

211

20

10

1

1

xwxw

xwxwT

T

24

SVM (linear) classifier

Minimize

Subject to

This is because minimizing

the margin is maximised

0)( wxwxg T

2

2

1)( wwJ

2for 1

for 1

ii

iii

x,y

,x,y

Niwxwy iT

i ,...,2,1 ,1)( 0

w

w

2

25

The above is a quadratic optimization task subject to a set of linear inequality constraints. The Karush-Kuhh-Tucker conditions, state that the minimizer satisfies:

• (1)

• (2)

• (3)

• (4)

• Where is the Lagrangian

0) ,L( 0 ,www

0) , , 00

ww(Lw

N,...,2,1i0i ,

Niwxwy iT

ii ,...,2,1 ,01)( 0

(.,.,.)L

)]([2

1) , ,( 0

10 wxwywwwwL i

Ti

N

ii

T

26

The solution: from the above, it turns out that

•

•

ii

N

ii xyw

1

01

i

N

ii y

27

Remarks:• The Lagrange multipliers can be either zero or

positive. Thus,

–

where , corresponding to positive Lagrange multipliers

– From constraint (4) above, i.e.,

the vectors contributing to

ii

N

ii xyw

s

1

0NN s

Niwxwy iT

ii ,...,2,1 ,0]1)([ 0

10 wxw iT

wsatisfy

28

– These vectors are known as SUPPORT VECTORS and are the closest vectors, from each class, to the classifier.

– Once is computed, is determined from conditions (4).

– The optimal hyperplane classifier of a support vector machine is UNIQUE.

w 0w

29

Dual Problem Formulation• The SVM formulation is a convex

programming problem, with– Convex cost function– Convex region of feasible solutions

• Thus, its solution can be achieved by its dual problem, i.e.,

– maximize

– subject to

),,( 0 wwL

0

01

1

i

N

ii

ii

N

ii

y

xyw

λ

30

• Combine the above to obtain

– maximize

– subject to

)2

1(

1j

Tijij

iji

N

ii xxyy

0

01

i

N

ii y

λ

31

Remarks:• Support vectors enter via inner products• Although the solution, w, is unique, the

Lagrange multipliers ARE NOT.

Non-Separable classes

32

In this case there is no hyperplane, such that

• Recall that the margin is defined as twice the distance between the following two hyperplanes

xwxwT ,1)(0

1

and

1

0

0

wxw

wxw

T

T

33

The training vectors belong to one of three possible categories

• A. Vectors outside the band which are correctly classified, i.e.,

• B. Vectors inside the band, and correctly classified, i.e.,

• C. Vectors misclassified, i.e.

1)( 0 wxwy Ti

1)(0 0 wxwy Ti

0)( 0 wxwy Ti

34

All three cases above can be represented as

A.B.C.

are known as slack variables

iT

i wxwy 1)( 0

i

i

i

1

10

0

i

35

The goal of the optimization is now two-fold• Maximize margin• Minimize the number of patterns with ,

That is

where C a constant and

• I(.) not differentiable. In practice, we use an approximation

•

• Following similar procedure as before we obtain

0i

N

iiICwwwJ

1

2

0 2

1) , ,(

00

01)(

i

iiI

N

iiCwwwJ

1

2

0 2

1) , ,(

36

KKT conditions

Ni

Ni

Niwxwy

NiC

yλ

xyλw

ii

ii

iiT

ii

ii

i

N

ii

ii

N

ii

,...,2,1 ,0, )6(

,...,2,1 ,0 )5(

,...,2,1 ,0]1)([ )4(

,...,2,1 ,0 )3(

0 )2(

)1(

0

1

1

37

The associated dual problem

Maximize

subject to

Remarks: The only difference with the separable

class case is the existence of C in the constraints

)2

1(

1 ,j

Tijij

N

i jiii xxyy

N

1iii

i

0y

N,...,2,1i,C0