Kernel Methods: Support Vector Machines Maximum Margin Classifiers and Support Vector Machines.
Embed Size (px)
Transcript of Kernel Methods: Support Vector Machines Maximum Margin Classifiers and Support Vector Machines.
- Slide 1
Kernel Methods: Support Vector Machines Maximum Margin Classifiers and Support Vector Machines Slide 2 Generalized Linear Discriminant Functions A linear discriminant function g(x) can be written as: g(x) = wo + i w i x i i = 1, , d (d is number of features). We could add additional terms to obtain a quadratic function: g(x) = wo + i w i x i + i j w ij x i x j The quadratic discriminant function introduces d(d-1)/2 coefficients corresponding to the products of attributes. The surfaces are thus more complicated(hyperquadric surfaces). Slide 3 Generalized Linear Discriminant Functions We could even add more terms w ijk x i x j x k and obtain the class of polynomial discriminant functions. The generalized form is g(x) = i a i i (x) g(x) = a t Where the summation goes over all functions i (x). The i (x) functions are called the phi or functions. The function is now linear on the i (x). The functions map a d-dimensional x-space into a d dimensional y-space. Example: g(x) = a 1 + a 2 x + a 3 x 2 = (1 x x 2 ) t Slide 4 Figure 5.5 Slide 5 Historical Background Vladimir Vapnik: Publications: 6 books and over a hundred research papers. Developed a theory for expected risk minimization. Invented Support Vector Machines Slide 6 Historical Background Alexey Chervonenkis With Vladimir Vapnik developed the concept of the Vapnik-Chervonenkis dimension. Slide 7 Support Vector Machines What are support vector machines (SVMs)? A very popular classifier that is based on the concepts previously discussed on linear discriminants and the new concept of margins. To begin, SVMs preprocess the data by representing all examples in a higher dimensional space. With sufficiently high dimensions the classes can be separated by a hyperplane. Slide 8 The Margin Slide 9 The Goal in Support Vector Machines Now, let t be 1 or 1 depending on the example x being of class positive or negative. A separating hyperplane ensures that: t g(x) >= 0 The goal in support vector machines is to find the separating hyperplane with the largest margin. Margin is the distance between the hyperplane and the closest example to it. Slide 10 The Support Vectors Now the distance from a pattern x to a hyperplane is g(x) / ||w||. So lets change our objective to finding a vector w that maximizes the margin m in the equation: t g(x) / ||w|| >= m We can also say that the support vectors are those patterns x for which t g(x) / ||w|| = 1, because we can rescale the w vector and leave the hyperplane in the same place. Support vectors are equally close to the hyperplane. These are the patterns that are most difficult to separate. These are the most informative patterns. Slide 11 The Support Vectors We said we want to find a vector w that maximizes the equation: t g(x) / ||w|| >= 1 This means all we really need to do is to maximize ||w|| -1 under certain constraints. So we have the following optimization problem: arg min w ||w|| 2 subject to t g(x) >= 1 This can be solved using Lagrange Multipliers Slide 12 The Support Vectors What happens when there are unavoidable errors? arg min w ||w|| 2 + e i subject to t g(x i ) >= 1 - e i where e i is the error incurred by example x i These are known as slack variables. Slide 13 The Support Vectors We can write this in a dual form (Karush-Kuhn-Tucker construction). max i i j t i t j (x i. x j ) subject to 0