Post on 16-Apr-2017
Neural NetworksRadial Basis Functions Networks
Andres Mendez-Vazquez
December 10, 2015
1 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
2 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
3 / 96
Introduction
ObservationThe back-propagation algorithm for the design of a multilayer perceptronas described in the previous chapter may be viewed as the application of arecursive technique known in statistics as stochastic approximation.
NowWe take a completely different approach by viewing the design of a neuralnetwork as a curve fitting (approximation) problem in a high-dimensionalspace.
ThusLearning is equivalent to finding a surface in a multidimensional space thatprovides a best fit to the training data.
Under a statistical metric
4 / 96
Introduction
ObservationThe back-propagation algorithm for the design of a multilayer perceptronas described in the previous chapter may be viewed as the application of arecursive technique known in statistics as stochastic approximation.
NowWe take a completely different approach by viewing the design of a neuralnetwork as a curve fitting (approximation) problem in a high-dimensionalspace.
ThusLearning is equivalent to finding a surface in a multidimensional space thatprovides a best fit to the training data.
Under a statistical metric
4 / 96
Introduction
ObservationThe back-propagation algorithm for the design of a multilayer perceptronas described in the previous chapter may be viewed as the application of arecursive technique known in statistics as stochastic approximation.
NowWe take a completely different approach by viewing the design of a neuralnetwork as a curve fitting (approximation) problem in a high-dimensionalspace.
ThusLearning is equivalent to finding a surface in a multidimensional space thatprovides a best fit to the training data.
Under a statistical metric
4 / 96
Thus
In the context of a neural networkThe hidden units provide a set of "functions"
A "basis" for the input patterns when they are expanded into thehidden space.
Name of these functionsRadial-Basis functions.
5 / 96
Thus
In the context of a neural networkThe hidden units provide a set of "functions"
A "basis" for the input patterns when they are expanded into thehidden space.
Name of these functionsRadial-Basis functions.
5 / 96
History
These functions were first introducedAs the solution of the real multivariate interpolation problem
Right nowIt is now one of the main fields of research in numerical analysis.
6 / 96
History
These functions were first introducedAs the solution of the real multivariate interpolation problem
Right nowIt is now one of the main fields of research in numerical analysis.
6 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
7 / 96
A Basic Structure
We have the following structure1 Input Layer to connect with the environment.2 Hidden Layer applying a non-linear transformation.3 Output Layer applying a linear transformation.
Example
8 / 96
A Basic Structure
We have the following structure1 Input Layer to connect with the environment.2 Hidden Layer applying a non-linear transformation.3 Output Layer applying a linear transformation.
Example
8 / 96
A Basic Structure
We have the following structure1 Input Layer to connect with the environment.2 Hidden Layer applying a non-linear transformation.3 Output Layer applying a linear transformation.
Example
8 / 96
A Basic Structure
We have the following structure1 Input Layer to connect with the environment.2 Hidden Layer applying a non-linear transformation.3 Output Layer applying a linear transformation.
ExampleInput Nodes
Nonlinear Nodes
Linear Node
8 / 96
Why the non-linear transformation?
The justificationIn a paper by Cover (1965), a pattern-classification problem mapped to ahigh dimensional space is more likely to be linearly separable than in alow-dimensional space.
ThusA good reason to make the dimension in the hidden space in aRadial-Basis Function (RBF) network high
9 / 96
Why the non-linear transformation?
The justificationIn a paper by Cover (1965), a pattern-classification problem mapped to ahigh dimensional space is more likely to be linearly separable than in alow-dimensional space.
ThusA good reason to make the dimension in the hidden space in aRadial-Basis Function (RBF) network high
9 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
10 / 96
Cover’s Theorem
The Resumed StatementA complex pattern-classification problem cast in a high-dimensional spacenonlinearly is more likely to be linearly separable than in a low-dimensionalspace.
ActuallyIt is quite more complex...
11 / 96
Cover’s Theorem
The Resumed StatementA complex pattern-classification problem cast in a high-dimensional spacenonlinearly is more likely to be linearly separable than in a low-dimensionalspace.
ActuallyIt is quite more complex...
11 / 96
Some facts
A factOnce we know a set of patterns are linearly separable, the problem is easyto solve.
ConsiderA family of surfaces that separate the space in two regions.
In additionWe have a set of patterns
H = {x1,x2, ...,xN} (1)
12 / 96
Some facts
A factOnce we know a set of patterns are linearly separable, the problem is easyto solve.
ConsiderA family of surfaces that separate the space in two regions.
In additionWe have a set of patterns
H = {x1,x2, ...,xN} (1)
12 / 96
Some facts
A factOnce we know a set of patterns are linearly separable, the problem is easyto solve.
ConsiderA family of surfaces that separate the space in two regions.
In additionWe have a set of patterns
H = {x1,x2, ...,xN} (1)
12 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
13 / 96
Dichotomy (Binary Partition)
NowThe pattern set is split into two classes H1 and H2.
DefinitionA dichotomy (binary partition) of the points is said to be separable withrespect to the family of surfaces if a surface exists in the family thatseparates the points in the class H1 from those in the class H2.
DefineFor each pattern x ∈ H, we define a set of real valued measurementfunctions {φ1 (x) , φ2 (x) , ..., φd1 (x)}
14 / 96
Dichotomy (Binary Partition)
NowThe pattern set is split into two classes H1 and H2.
DefinitionA dichotomy (binary partition) of the points is said to be separable withrespect to the family of surfaces if a surface exists in the family thatseparates the points in the class H1 from those in the class H2.
DefineFor each pattern x ∈ H, we define a set of real valued measurementfunctions {φ1 (x) , φ2 (x) , ..., φd1 (x)}
14 / 96
Dichotomy (Binary Partition)
NowThe pattern set is split into two classes H1 and H2.
DefinitionA dichotomy (binary partition) of the points is said to be separable withrespect to the family of surfaces if a surface exists in the family thatseparates the points in the class H1 from those in the class H2.
DefineFor each pattern x ∈ H, we define a set of real valued measurementfunctions {φ1 (x) , φ2 (x) , ..., φd1 (x)}
14 / 96
Thus
We define the following function (Vector of measurements)
φ : H → Rd1 (2)
Defined as
φ (x) = (φ1 (x) , φ2 (x) , ..., φd1 (x))T (3)
NowSuppose that the pattern x is a vector in an d0-dimensional input space.
15 / 96
Thus
We define the following function (Vector of measurements)
φ : H → Rd1 (2)
Defined as
φ (x) = (φ1 (x) , φ2 (x) , ..., φd1 (x))T (3)
NowSuppose that the pattern x is a vector in an d0-dimensional input space.
15 / 96
Thus
We define the following function (Vector of measurements)
φ : H → Rd1 (2)
Defined as
φ (x) = (φ1 (x) , φ2 (x) , ..., φd1 (x))T (3)
NowSuppose that the pattern x is a vector in an d0-dimensional input space.
15 / 96
Then...
We have that the mapping φ (x)It maps points in d0-dimensional space into corresponding points in a newspace of dimension d1.
Each of this functions φi (x)It is known as a hidden function because it plays a role similar to thehidden unit in a feed-forward neural network.
ThusWe have that the space spanned by the set of hidden functions{φi (x)}d1
i=1 is called as the hidden space of feature space.
16 / 96
Then...
We have that the mapping φ (x)It maps points in d0-dimensional space into corresponding points in a newspace of dimension d1.
Each of this functions φi (x)It is known as a hidden function because it plays a role similar to thehidden unit in a feed-forward neural network.
ThusWe have that the space spanned by the set of hidden functions{φi (x)}d1
i=1 is called as the hidden space of feature space.
16 / 96
Then...
We have that the mapping φ (x)It maps points in d0-dimensional space into corresponding points in a newspace of dimension d1.
Each of this functions φi (x)It is known as a hidden function because it plays a role similar to thehidden unit in a feed-forward neural network.
ThusWe have that the space spanned by the set of hidden functions{φi (x)}d1
i=1 is called as the hidden space of feature space.
16 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
17 / 96
φ-separable functionsDefinitionA dichotomy {H1,H2} of H is said to be φ-separable if there exists ad1-dimensional vector w such that
1 wTφ (x) > 0 if x ∈ H1.2 wTφ (x) < 0 if x ∈ H2.
Clearly the hyperplane is defined by the equation
wTφ (x) = 0 (4)
NowThe inverse image of this hyperplane
Hyp−1 ={
x|wTφ (x) = 0}
(5)
define the separating surface in the input space.18 / 96
φ-separable functionsDefinitionA dichotomy {H1,H2} of H is said to be φ-separable if there exists ad1-dimensional vector w such that
1 wTφ (x) > 0 if x ∈ H1.2 wTφ (x) < 0 if x ∈ H2.
Clearly the hyperplane is defined by the equation
wTφ (x) = 0 (4)
NowThe inverse image of this hyperplane
Hyp−1 ={
x|wTφ (x) = 0}
(5)
define the separating surface in the input space.18 / 96
φ-separable functionsDefinitionA dichotomy {H1,H2} of H is said to be φ-separable if there exists ad1-dimensional vector w such that
1 wTφ (x) > 0 if x ∈ H1.2 wTφ (x) < 0 if x ∈ H2.
Clearly the hyperplane is defined by the equation
wTφ (x) = 0 (4)
NowThe inverse image of this hyperplane
Hyp−1 ={
x|wTφ (x) = 0}
(5)
define the separating surface in the input space.18 / 96
φ-separable functionsDefinitionA dichotomy {H1,H2} of H is said to be φ-separable if there exists ad1-dimensional vector w such that
1 wTφ (x) > 0 if x ∈ H1.2 wTφ (x) < 0 if x ∈ H2.
Clearly the hyperplane is defined by the equation
wTφ (x) = 0 (4)
NowThe inverse image of this hyperplane
Hyp−1 ={
x|wTφ (x) = 0}
(5)
define the separating surface in the input space.18 / 96
Now
Taking in considerationA natural class of mappings obtained by using a linear combination ofr-wise products of the pattern vector coordinates.
They are calledAs the rth-order rational varieties.
A rational variety of order r in dimensional d0 is described by∑0≤i1≤i2≤...≤ir≤d0
ai1i2...ir xi1xi2 ...xir = 0 (6)
where xi is the ith coordinate of the input vector x and x0 is set to unityin order to express the previous equation in homogeneous form.
19 / 96
Now
Taking in considerationA natural class of mappings obtained by using a linear combination ofr-wise products of the pattern vector coordinates.
They are calledAs the rth-order rational varieties.
A rational variety of order r in dimensional d0 is described by∑0≤i1≤i2≤...≤ir≤d0
ai1i2...ir xi1xi2 ...xir = 0 (6)
where xi is the ith coordinate of the input vector x and x0 is set to unityin order to express the previous equation in homogeneous form.
19 / 96
Now
Taking in considerationA natural class of mappings obtained by using a linear combination ofr-wise products of the pattern vector coordinates.
They are calledAs the rth-order rational varieties.
A rational variety of order r in dimensional d0 is described by∑0≤i1≤i2≤...≤ir≤d0
ai1i2...ir xi1xi2 ...xir = 0 (6)
where xi is the ith coordinate of the input vector x and x0 is set to unityin order to express the previous equation in homogeneous form.
19 / 96
Now
Taking in considerationA natural class of mappings obtained by using a linear combination ofr-wise products of the pattern vector coordinates.
They are calledAs the rth-order rational varieties.
A rational variety of order r in dimensional d0 is described by∑0≤i1≤i2≤...≤ir≤d0
ai1i2...ir xi1xi2 ...xir = 0 (6)
where xi is the ith coordinate of the input vector x and x0 is set to unityin order to express the previous equation in homogeneous form.
19 / 96
Homogenous Functions
DefinitionA function f (x) is said to be homogeneous of degree n if, by introducing aconstant parameter λ, replacing the variable x with λx we find:
f (λx) = λnf (x) (7)
20 / 96
Homogeneous Equation
Equation (Eq. 6)A rth order product of entries xi of x, xi1xi2 ...xir , is called a monomial
PropertiesFor an input space of dimensionality d0, there are(
d0r
)= d0!
(d0 − r)!r ! (8)
monomials in (Eq. 6).
21 / 96
Homogeneous Equation
Equation (Eq. 6)A rth order product of entries xi of x, xi1xi2 ...xir , is called a monomial
PropertiesFor an input space of dimensionality d0, there are(
d0r
)= d0!
(d0 − r)!r ! (8)
monomials in (Eq. 6).
21 / 96
Example of these surfaces
Hyperplanes (first-order rational varieties)
22 / 96
Example of these surfaces
Hyperplanes (first-order rational varieties)
23 / 96
Example of these surfaces
Quadrices (second-order rational varieties)
24 / 96
Example of these surfaces
Hyperspheres (quadrics with certain linear constraints on thecoefficients)
25 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
26 / 96
The Stochastic Experiment
SupposeYou have the following activation patterns x1,x2, ...,xN are chosenindependently.
SupposeThat all possible dichotomies of H = {x1,x2, ...,xN} are equiprobable.
Now given P (N , d1) the probability that a particular dichotomypicked at random is φ-separable
P (N , d1) =(1
2
)N−1 d1−1∑m=0
(N − 1
m
)(9)
27 / 96
The Stochastic Experiment
SupposeYou have the following activation patterns x1,x2, ...,xN are chosenindependently.
SupposeThat all possible dichotomies of H = {x1,x2, ...,xN} are equiprobable.
Now given P (N , d1) the probability that a particular dichotomypicked at random is φ-separable
P (N , d1) =(1
2
)N−1 d1−1∑m=0
(N − 1
m
)(9)
27 / 96
The Stochastic Experiment
SupposeYou have the following activation patterns x1,x2, ...,xN are chosenindependently.
SupposeThat all possible dichotomies of H = {x1,x2, ...,xN} are equiprobable.
Now given P (N , d1) the probability that a particular dichotomypicked at random is φ-separable
P (N , d1) =(1
2
)N−1 d1−1∑m=0
(N − 1
m
)(9)
27 / 96
What?
Basically (Eq. 9) representsThe essence of Cover’s Separability Theorem.
Something NotableIt is a statement of the fact that the cumulative binomial distributioncorresponding to the probability that N − 1 (Flips of a coin) samples willbe separable in a mapping of d1 − 1 (heads) or fewer dimensions.
SpecificallyThe higher we make the hidden space in the radial basis function thecloser is the probability of P (N , d1) to one.
28 / 96
What?
Basically (Eq. 9) representsThe essence of Cover’s Separability Theorem.
Something NotableIt is a statement of the fact that the cumulative binomial distributioncorresponding to the probability that N − 1 (Flips of a coin) samples willbe separable in a mapping of d1 − 1 (heads) or fewer dimensions.
SpecificallyThe higher we make the hidden space in the radial basis function thecloser is the probability of P (N , d1) to one.
28 / 96
What?
Basically (Eq. 9) representsThe essence of Cover’s Separability Theorem.
Something NotableIt is a statement of the fact that the cumulative binomial distributioncorresponding to the probability that N − 1 (Flips of a coin) samples willbe separable in a mapping of d1 − 1 (heads) or fewer dimensions.
SpecificallyThe higher we make the hidden space in the radial basis function thecloser is the probability of P (N , d1) to one.
28 / 96
Final ingredients if the Cover’s Theorem
FirstNonlinear formulation of the hidden function defined by φ (x), where x isthe input vector and i = 1, 2, ..., d1.
SecondHigh dimensionality of the hidden space compared to the input space.This dimensionality is determined by the value assigned to d_1 (i.e.,the number of hidden units).
ThenIn general, a complex pattern-classification problem cast inhighdimensional space nonlinearly is more likely to be linearly separablethan in a lowdimensional space.
29 / 96
Final ingredients if the Cover’s Theorem
FirstNonlinear formulation of the hidden function defined by φ (x), where x isthe input vector and i = 1, 2, ..., d1.
SecondHigh dimensionality of the hidden space compared to the input space.This dimensionality is determined by the value assigned to d_1 (i.e.,the number of hidden units).
ThenIn general, a complex pattern-classification problem cast inhighdimensional space nonlinearly is more likely to be linearly separablethan in a lowdimensional space.
29 / 96
Final ingredients if the Cover’s Theorem
FirstNonlinear formulation of the hidden function defined by φ (x), where x isthe input vector and i = 1, 2, ..., d1.
SecondHigh dimensionality of the hidden space compared to the input space.This dimensionality is determined by the value assigned to d_1 (i.e.,the number of hidden units).
ThenIn general, a complex pattern-classification problem cast inhighdimensional space nonlinearly is more likely to be linearly separablethan in a lowdimensional space.
29 / 96
Final ingredients if the Cover’s Theorem
FirstNonlinear formulation of the hidden function defined by φ (x), where x isthe input vector and i = 1, 2, ..., d1.
SecondHigh dimensionality of the hidden space compared to the input space.This dimensionality is determined by the value assigned to d_1 (i.e.,the number of hidden units).
ThenIn general, a complex pattern-classification problem cast inhighdimensional space nonlinearly is more likely to be linearly separablethan in a lowdimensional space.
29 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
30 / 96
There is always an exception to every rule!!!
The XOR Problem
0
1
1
Class 1
Class 2
31 / 96
Now
We define the following radial functions
φ1 (x) = exp{‖x − t1‖22
}where t1 = (1, 1)T
φ2 (x) = exp{‖x − t2‖22
}where t2 = (1, 1)T
ThenIf we apply our classic mapping φ (x) = [φ1 (x) , φ2 (x)]:
Original Mapping(0, 1) → (0.3678, 0.3678)(1, 0) → (0.3678, 0.3678)(0, 0) → (0.1353, 1)(1, 1) → (1, 0.1353)
32 / 96
Now
We define the following radial functions
φ1 (x) = exp{‖x − t1‖22
}where t1 = (1, 1)T
φ2 (x) = exp{‖x − t2‖22
}where t2 = (1, 1)T
ThenIf we apply our classic mapping φ (x) = [φ1 (x) , φ2 (x)]:
Original Mapping(0, 1) → (0.3678, 0.3678)(1, 0) → (0.3678, 0.3678)(0, 0) → (0.1353, 1)(1, 1) → (1, 0.1353)
32 / 96
New Space
We have the following new φ1 − φ2 space
0
1
1
Class 1
Class 2
33 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
34 / 96
Separating Capacity of a Surface
Something Notable(Eq. 9) has an important bearing on the expected maximum number ofrandomly assigned patterns that are linearly separable in amultidimensional space.
Now, given our patterns {x i}Ni=1
Given N be a random variable defined as the largest integer such that thesequence is φ-separable.
We have that
Prob (N = n) = P (n, d1)− P (n + 1, d1) (10)
35 / 96
Separating Capacity of a Surface
Something Notable(Eq. 9) has an important bearing on the expected maximum number ofrandomly assigned patterns that are linearly separable in amultidimensional space.
Now, given our patterns {x i}Ni=1
Given N be a random variable defined as the largest integer such that thesequence is φ-separable.
We have that
Prob (N = n) = P (n, d1)− P (n + 1, d1) (10)
35 / 96
Separating Capacity of a Surface
Something Notable(Eq. 9) has an important bearing on the expected maximum number ofrandomly assigned patterns that are linearly separable in amultidimensional space.
Now, given our patterns {x i}Ni=1
Given N be a random variable defined as the largest integer such that thesequence is φ-separable.
We have that
Prob (N = n) = P (n, d1)− P (n + 1, d1) (10)
35 / 96
Separating Capacity of a Surface
Then
Prob (N = n) =(1
2
)n(
n − 1d1 − 1
), n = 0, 1, 2... (11)
Remark:(
nd1
)=(
n − 1d1 − 1
)+(
n − 1d1
), 0 < d1 < n
To interpret thisRecall the negative binomial distribution.
It is a repeated sequence of Bernoulli TrialsWith k failures preceding the rth success.
36 / 96
Separating Capacity of a Surface
Then
Prob (N = n) =(1
2
)n(
n − 1d1 − 1
), n = 0, 1, 2... (11)
Remark:(
nd1
)=(
n − 1d1 − 1
)+(
n − 1d1
), 0 < d1 < n
To interpret thisRecall the negative binomial distribution.
It is a repeated sequence of Bernoulli TrialsWith k failures preceding the rth success.
36 / 96
Separating Capacity of a Surface
Then
Prob (N = n) =(1
2
)n(
n − 1d1 − 1
), n = 0, 1, 2... (11)
Remark:(
nd1
)=(
n − 1d1 − 1
)+(
n − 1d1
), 0 < d1 < n
To interpret thisRecall the negative binomial distribution.
It is a repeated sequence of Bernoulli TrialsWith k failures preceding the rth success.
36 / 96
Separating Capacity of a Surface
Thus, we have thatGiven p and q the probabilities of success and failure, respectively, withp + q = 1.
Definition
p (K = k|p, q) =(
r + k − 1k
)prqk (12)
What happened with p = q = 12 and k + r = n
Any idea?
37 / 96
Separating Capacity of a Surface
Thus, we have thatGiven p and q the probabilities of success and failure, respectively, withp + q = 1.
Definition
p (K = k|p, q) =(
r + k − 1k
)prqk (12)
What happened with p = q = 12 and k + r = n
Any idea?
37 / 96
Separating Capacity of a Surface
Thus, we have thatGiven p and q the probabilities of success and failure, respectively, withp + q = 1.
Definition
p (K = k|p, q) =(
r + k − 1k
)prqk (12)
What happened with p = q = 12 and k + r = n
Any idea?
37 / 96
Separating Capacity of a Surface
Thus(Eq. 11) is just the negative binomial distribution shifted d1 units to theright with parameters d1 and 1
2
FinallyN corresponds to thew “waiting time” for d1 th failure in a sequence oftosses of a fair coin.
We have then
E [N ] = 2d1
Median [N ] = 2d1
38 / 96
Separating Capacity of a Surface
Thus(Eq. 11) is just the negative binomial distribution shifted d1 units to theright with parameters d1 and 1
2
FinallyN corresponds to thew “waiting time” for d1 th failure in a sequence oftosses of a fair coin.
We have then
E [N ] = 2d1
Median [N ] = 2d1
38 / 96
Separating Capacity of a Surface
Thus(Eq. 11) is just the negative binomial distribution shifted d1 units to theright with parameters d1 and 1
2
FinallyN corresponds to thew “waiting time” for d1 th failure in a sequence oftosses of a fair coin.
We have then
E [N ] = 2d1
Median [N ] = 2d1
38 / 96
This allows to define the Corollary to Cover’s Theorem
A celebrated asymptotic resultThe expected maximum number of randomly assigned patterns (vectors)that are linearly separable in a space of dimensionality d1 is equal to 2d1 .
Something Notable
This result suggests that 2d1 is a natural definition of the separatingcapacity of a family of decision surfaces having d1 degrees of freedom.
39 / 96
This allows to define the Corollary to Cover’s Theorem
A celebrated asymptotic resultThe expected maximum number of randomly assigned patterns (vectors)that are linearly separable in a space of dimensionality d1 is equal to 2d1 .
Something Notable
This result suggests that 2d1 is a natural definition of the separatingcapacity of a family of decision surfaces having d1 degrees of freedom.
39 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
40 / 96
Given a problem of non-linearly separable patterns
It is possible to see thatThere is a benefit to be gained by mapping the input space into a newspace of high enough dimension
For this, we use a non-linear mapQuite similar to solve a difficult non-linear filtering problem by mapping itto high dimension, then solving it as a linear filtering problem.
41 / 96
Given a problem of non-linearly separable patterns
It is possible to see thatThere is a benefit to be gained by mapping the input space into a newspace of high enough dimension
For this, we use a non-linear mapQuite similar to solve a difficult non-linear filtering problem by mapping itto high dimension, then solving it as a linear filtering problem.
41 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
42 / 96
Take in consideration the following architecture
Mapping from input space to hidden space, followed by a linearmapping to output space!!!
Input NodesNonlinear Nodes
Linear Node
43 / 96
This can be seen as
We have the following map
s : Rd0 → R (13)
ThereforeWe may think of s as a hypersurface (graph) Γ ⊂ Rd0+1
44 / 96
This can be seen as
We have the following map
s : Rd0 → R (13)
ThereforeWe may think of s as a hypersurface (graph) Γ ⊂ Rd0+1
44 / 96
ExampleWe have that the Red planes represent the mappings and the Gray isthe Linear Separator
45 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
46 / 96
General Idea
FirstThe training phase constitutes the optimization of a fitting procedurefor the surface Γ.It is based in the know data points as input-output patterns.
SecondThe generalization phase is synonymous with interpolation betweenthe data points.The interpolation being performed along the constrained surfacegenerated by the fitting procedure.
47 / 96
General Idea
FirstThe training phase constitutes the optimization of a fitting procedurefor the surface Γ.It is based in the know data points as input-output patterns.
SecondThe generalization phase is synonymous with interpolation betweenthe data points.The interpolation being performed along the constrained surfacegenerated by the fitting procedure.
47 / 96
General Idea
FirstThe training phase constitutes the optimization of a fitting procedurefor the surface Γ.It is based in the know data points as input-output patterns.
SecondThe generalization phase is synonymous with interpolation betweenthe data points.The interpolation being performed along the constrained surfacegenerated by the fitting procedure.
47 / 96
General Idea
FirstThe training phase constitutes the optimization of a fitting procedurefor the surface Γ.It is based in the know data points as input-output patterns.
SecondThe generalization phase is synonymous with interpolation betweenthe data points.The interpolation being performed along the constrained surfacegenerated by the fitting procedure.
47 / 96
This leads to the theory of multi-variable interpolation
Interpolation ProblemGiven a set of N different points
{xi ∈ Rd0 |i = 1, 2, ...,N
}and a
corresponding set of N real numbers{di ∈ R1|i = 1, 2, ...,N
}, find a
function F : RN → R that satisfies the interpolation condition:
F (xi) = di i = 1, 2, ...,N (14)
RemarkFor strict interpolation as specified here, the interpolating surface isconstrained to pass through all the training data points.
48 / 96
This leads to the theory of multi-variable interpolation
Interpolation ProblemGiven a set of N different points
{xi ∈ Rd0 |i = 1, 2, ...,N
}and a
corresponding set of N real numbers{di ∈ R1|i = 1, 2, ...,N
}, find a
function F : RN → R that satisfies the interpolation condition:
F (xi) = di i = 1, 2, ...,N (14)
RemarkFor strict interpolation as specified here, the interpolating surface isconstrained to pass through all the training data points.
48 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
49 / 96
Radial-Basis Functions (RBF)
The function F has the following form (Powell, 1988)
F (x) =N∑
i=1wiφ (‖x − xi‖) (15)
Where
{φ (‖x − xi‖) |i = 1, ...,N}
is a set of N arbitrary, generally non-linear, functions, know as RBF with‖·‖ denotes a norm that is usually Euclidean.
In additionThe know data points xi ∈ Rd0 i = 1, 2, ...,N are taken to be the centersof the radial basis functions.
50 / 96
Radial-Basis Functions (RBF)
The function F has the following form (Powell, 1988)
F (x) =N∑
i=1wiφ (‖x − xi‖) (15)
Where
{φ (‖x − xi‖) |i = 1, ...,N}
is a set of N arbitrary, generally non-linear, functions, know as RBF with‖·‖ denotes a norm that is usually Euclidean.
In additionThe know data points xi ∈ Rd0 i = 1, 2, ...,N are taken to be the centersof the radial basis functions.
50 / 96
Radial-Basis Functions (RBF)
The function F has the following form (Powell, 1988)
F (x) =N∑
i=1wiφ (‖x − xi‖) (15)
Where
{φ (‖x − xi‖) |i = 1, ...,N}
is a set of N arbitrary, generally non-linear, functions, know as RBF with‖·‖ denotes a norm that is usually Euclidean.
In additionThe know data points xi ∈ Rd0 i = 1, 2, ...,N are taken to be the centersof the radial basis functions.
50 / 96
A Set of Simultaneous Linear Equations
Given
φji = φ (‖xj − xi‖) , (j, i) = 1, 2, ...,N (16)
Using (Eq. 14) and (Eq. 15), we getφ11 φ12 · · · φ1Nφ21 φ22 · · · φ2N...
......
...φN1 φN2 · · · φNN
w1w2...
wN
=
d1d2...
dN
(17)
51 / 96
A Set of Simultaneous Linear Equations
Given
φji = φ (‖xj − xi‖) , (j, i) = 1, 2, ...,N (16)
Using (Eq. 14) and (Eq. 15), we getφ11 φ12 · · · φ1Nφ21 φ22 · · · φ2N...
......
...φN1 φN2 · · · φNN
w1w2...
wN
=
d1d2...
dN
(17)
51 / 96
Now
We can create the following vectorsd = [d1, d2, ..., dN ]T (Response vector).w = [w1,w2, ...,wN ]T (Linear weight vector).
Now, we define a N × N matrix called interpolation matrix
Φ = {φji | (j, i) = 1, 2, ...,N} (18)
Thus, we have
Φw = x (19)
52 / 96
Now
We can create the following vectorsd = [d1, d2, ..., dN ]T (Response vector).w = [w1,w2, ...,wN ]T (Linear weight vector).
Now, we define a N × N matrix called interpolation matrix
Φ = {φji | (j, i) = 1, 2, ...,N} (18)
Thus, we have
Φw = x (19)
52 / 96
Now
We can create the following vectorsd = [d1, d2, ..., dN ]T (Response vector).w = [w1,w2, ...,wN ]T (Linear weight vector).
Now, we define a N × N matrix called interpolation matrix
Φ = {φji | (j, i) = 1, 2, ...,N} (18)
Thus, we have
Φw = x (19)
52 / 96
From here
Assuming that Φ is a non-singular matrix
w = Φ−1x (20)
QuestionHow can we be sure that the interpolation matrix Φ is non-singular?
AnswerIt turns out that for a large class of radial-basis functions and undercertain conditions the non-singularity happens!!!
53 / 96
From here
Assuming that Φ is a non-singular matrix
w = Φ−1x (20)
QuestionHow can we be sure that the interpolation matrix Φ is non-singular?
AnswerIt turns out that for a large class of radial-basis functions and undercertain conditions the non-singularity happens!!!
53 / 96
From here
Assuming that Φ is a non-singular matrix
w = Φ−1x (20)
QuestionHow can we be sure that the interpolation matrix Φ is non-singular?
AnswerIt turns out that for a large class of radial-basis functions and undercertain conditions the non-singularity happens!!!
53 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
54 / 96
Introduction
ObservationThe strict interpolation procedure described may not be a good strategyfor the training of RBF networks for certain classes of tasks.
ReasonIf the number of data points is much larger than the number of degrees offreedom of the underlying physical process.
ThusThe network may end up fitting misleading variations due to idiosyncrasiesor noise in the input data.
55 / 96
Introduction
ObservationThe strict interpolation procedure described may not be a good strategyfor the training of RBF networks for certain classes of tasks.
ReasonIf the number of data points is much larger than the number of degrees offreedom of the underlying physical process.
ThusThe network may end up fitting misleading variations due to idiosyncrasiesor noise in the input data.
55 / 96
Introduction
ObservationThe strict interpolation procedure described may not be a good strategyfor the training of RBF networks for certain classes of tasks.
ReasonIf the number of data points is much larger than the number of degrees offreedom of the underlying physical process.
ThusThe network may end up fitting misleading variations due to idiosyncrasiesor noise in the input data.
55 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
56 / 96
Well-posed
The ProblemAssume that we have a domain X and a range Y , metric spaces.
They are related by a mapping
f : X → Y (21)
DefinitionThe problem of reconstructing the mapping f is said to be well-posed ifthree conditions are satisfied: Existence, Uniqueness and Continuity.
57 / 96
Well-posed
The ProblemAssume that we have a domain X and a range Y , metric spaces.
They are related by a mapping
f : X → Y (21)
DefinitionThe problem of reconstructing the mapping f is said to be well-posed ifthree conditions are satisfied: Existence, Uniqueness and Continuity.
57 / 96
Well-posed
The ProblemAssume that we have a domain X and a range Y , metric spaces.
They are related by a mapping
f : X → Y (21)
DefinitionThe problem of reconstructing the mapping f is said to be well-posed ifthree conditions are satisfied: Existence, Uniqueness and Continuity.
57 / 96
Defining the meaning of this
ExistenceFor every input vector x ∈ X , there does exist an output y = f (x), wherey ∈ Y .
UniquenessFor any pair of input vectors x, t ∈ X , we have f (x) = f (t) if and only ifx = t.
ContinuityThe mapping is continuous, if for any ε > 0 exists δ such that thecondition dX (x, t) < δ implies dY (f (x) , f (t)) < ε.
58 / 96
Defining the meaning of this
ExistenceFor every input vector x ∈ X , there does exist an output y = f (x), wherey ∈ Y .
UniquenessFor any pair of input vectors x, t ∈ X , we have f (x) = f (t) if and only ifx = t.
ContinuityThe mapping is continuous, if for any ε > 0 exists δ such that thecondition dX (x, t) < δ implies dY (f (x) , f (t)) < ε.
58 / 96
Defining the meaning of this
ExistenceFor every input vector x ∈ X , there does exist an output y = f (x), wherey ∈ Y .
UniquenessFor any pair of input vectors x, t ∈ X , we have f (x) = f (t) if and only ifx = t.
ContinuityThe mapping is continuous, if for any ε > 0 exists δ such that thecondition dX (x, t) < δ implies dY (f (x) , f (t)) < ε.
58 / 96
Basically
ExampleMapping
59 / 96
Ill-Posed
ThereforeIf any of these conditions is not satisfied, the problem is said to beill-posed.
BasicallyAn ill-posed problem means that large data sets may contain asurprisingly small amount of information about the desired solution.
60 / 96
Ill-Posed
ThereforeIf any of these conditions is not satisfied, the problem is said to beill-posed.
BasicallyAn ill-posed problem means that large data sets may contain asurprisingly small amount of information about the desired solution.
60 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
61 / 96
Learning from data
Rebuilding the physical phenomena using the samplesPhysical Phenomenon
Data
62 / 96
We have the following
Physical PhenomenaSpeech, pictures, radar signals, sonar signals, seismic data.
It is a well-posed dataBut learning form such data i.e. rebuilding the hypersurface can be anill-posed inverse problem.
63 / 96
We have the following
Physical PhenomenaSpeech, pictures, radar signals, sonar signals, seismic data.
It is a well-posed dataBut learning form such data i.e. rebuilding the hypersurface can be anill-posed inverse problem.
63 / 96
Why
FirstThe existence criterion may be violated in that a distinct output may notexist for every input
SecondThere may not be as much information in the training sample as we reallyneed to reconstruct the input-output mapping uniquely.
ThirdThe unavoidable presence of noise or imprecision in real-life training dataadds uncertainty to the reconstructed input-output mapping.
64 / 96
Why
FirstThe existence criterion may be violated in that a distinct output may notexist for every input
SecondThere may not be as much information in the training sample as we reallyneed to reconstruct the input-output mapping uniquely.
ThirdThe unavoidable presence of noise or imprecision in real-life training dataadds uncertainty to the reconstructed input-output mapping.
64 / 96
Why
FirstThe existence criterion may be violated in that a distinct output may notexist for every input
SecondThere may not be as much information in the training sample as we reallyneed to reconstruct the input-output mapping uniquely.
ThirdThe unavoidable presence of noise or imprecision in real-life training dataadds uncertainty to the reconstructed input-output mapping.
64 / 96
The noise problem
Getting out of the range
Mapping+Noise
65 / 96
How?
This can happen whenThere is a lack of information!!!
Lanczos, 1964“A lack of information cannot be remedied by any mathematical trickery.”
66 / 96
How?
This can happen whenThere is a lack of information!!!
Lanczos, 1964“A lack of information cannot be remedied by any mathematical trickery.”
66 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
67 / 96
How do we solve the problem?
Something NotableIn 1963, Tikhonov proposed a new method called regularization for solvingill-posed ’
TikhonovHe was a Soviet and Russian mathematician known for importantcontributions to topology, functional analysis, mathematical physics, andill-posed problems.
68 / 96
How do we solve the problem?
Something NotableIn 1963, Tikhonov proposed a new method called regularization for solvingill-posed ’
TikhonovHe was a Soviet and Russian mathematician known for importantcontributions to topology, functional analysis, mathematical physics, andill-posed problems.
68 / 96
Also Known as Ridge Regression
SetupWe have:
Input Signal{
xi ∈ Rd0}N
i=1.
Output Signal {di ∈ R}Ni=1.
In additionNote that the output is assumed to be one-dimensional.
69 / 96
Also Known as Ridge Regression
SetupWe have:
Input Signal{
xi ∈ Rd0}N
i=1.
Output Signal {di ∈ R}Ni=1.
In additionNote that the output is assumed to be one-dimensional.
69 / 96
Now, assuming that you have an approximation functiony = F (x)
Standard Error Term
Es (F) = 12
N∑i=1
(di − yi) = 12
N∑i=1
(di − F (xi)) (22)
Regularization Term
Ec (F) = 12 ‖DF‖2 (23)
WhereD is a linear differential operator.
70 / 96
Now, assuming that you have an approximation functiony = F (x)
Standard Error Term
Es (F) = 12
N∑i=1
(di − yi) = 12
N∑i=1
(di − F (xi)) (22)
Regularization Term
Ec (F) = 12 ‖DF‖2 (23)
WhereD is a linear differential operator.
70 / 96
Now, assuming that you have an approximation functiony = F (x)
Standard Error Term
Es (F) = 12
N∑i=1
(di − yi) = 12
N∑i=1
(di − F (xi)) (22)
Regularization Term
Ec (F) = 12 ‖DF‖2 (23)
WhereD is a linear differential operator.
70 / 96
Now
Ordinarily y = F (x)Normally, the function space representing the functional F is the L2 spacethat consist of all real-valued functions f (x) with x ∈ Rd0
The quantity to be minimized in regularization theory is
E (f ) = 12
N∑i=1
(di − f (xi)) + 12 ‖Df ‖2 (24)
Whereλ is a positive real number called the regularization parameter.E (f ) is called the Tikhonov functional.
71 / 96
Now
Ordinarily y = F (x)Normally, the function space representing the functional F is the L2 spacethat consist of all real-valued functions f (x) with x ∈ Rd0
The quantity to be minimized in regularization theory is
E (f ) = 12
N∑i=1
(di − f (xi)) + 12 ‖Df ‖2 (24)
Whereλ is a positive real number called the regularization parameter.E (f ) is called the Tikhonov functional.
71 / 96
Now
Ordinarily y = F (x)Normally, the function space representing the functional F is the L2 spacethat consist of all real-valued functions f (x) with x ∈ Rd0
The quantity to be minimized in regularization theory is
E (f ) = 12
N∑i=1
(di − f (xi)) + 12 ‖Df ‖2 (24)
Whereλ is a positive real number called the regularization parameter.E (f ) is called the Tikhonov functional.
71 / 96
Now
Ordinarily y = F (x)Normally, the function space representing the functional F is the L2 spacethat consist of all real-valued functions f (x) with x ∈ Rd0
The quantity to be minimized in regularization theory is
E (f ) = 12
N∑i=1
(di − f (xi)) + 12 ‖Df ‖2 (24)
Whereλ is a positive real number called the regularization parameter.E (f ) is called the Tikhonov functional.
71 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
72 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Introduction
What did we see until now?The design of learning machines from two main points:
Statistical Point of ViewLinear Algebra and Optimization Point of View
Going back to the probability modelsWe might think in the machine to be learned as a function g (x|D)....
Something as curve fitting...
Under a data set
D = {(xi , yi) |i = 1, 2, ...,N} (25)
Remark: Where the xi ∼ p (x|Θ)!!!
73 / 96
Thus, we have that
Two main functionsA function g (x|D) obtained using some algorithm!!!E [y|x] the optimal regression...
ImportantThe key factor here is the dependence of the approximation on D.
Why?The approximation may be very good for a specific training data set butvery bad for another.
This is the reason of studying fusion of information at decision level...
74 / 96
Thus, we have that
Two main functionsA function g (x|D) obtained using some algorithm!!!E [y|x] the optimal regression...
ImportantThe key factor here is the dependence of the approximation on D.
Why?The approximation may be very good for a specific training data set butvery bad for another.
This is the reason of studying fusion of information at decision level...
74 / 96
Thus, we have that
Two main functionsA function g (x|D) obtained using some algorithm!!!E [y|x] the optimal regression...
ImportantThe key factor here is the dependence of the approximation on D.
Why?The approximation may be very good for a specific training data set butvery bad for another.
This is the reason of studying fusion of information at decision level...
74 / 96
Thus, we have that
Two main functionsA function g (x|D) obtained using some algorithm!!!E [y|x] the optimal regression...
ImportantThe key factor here is the dependence of the approximation on D.
Why?The approximation may be very good for a specific training data set butvery bad for another.
This is the reason of studying fusion of information at decision level...
74 / 96
Thus, we have that
Two main functionsA function g (x|D) obtained using some algorithm!!!E [y|x] the optimal regression...
ImportantThe key factor here is the dependence of the approximation on D.
Why?The approximation may be very good for a specific training data set butvery bad for another.
This is the reason of studying fusion of information at decision level...
74 / 96
How do we measure the difference
We have that
Var(X) = E((X − µ)2)
We can do that for our data
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2
)
Now, if we add and subtract
ED [g (x|D)] (26)
Remark: The expected output of the machine g (x|D)
75 / 96
How do we measure the difference
We have that
Var(X) = E((X − µ)2)
We can do that for our data
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2
)
Now, if we add and subtract
ED [g (x|D)] (26)
Remark: The expected output of the machine g (x|D)
75 / 96
How do we measure the difference
We have that
Var(X) = E((X − µ)2)
We can do that for our data
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2
)
Now, if we add and subtract
ED [g (x|D)] (26)
Remark: The expected output of the machine g (x|D)
75 / 96
How do we measure the difference
We have that
Var(X) = E((X − µ)2)
We can do that for our data
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2
)
Now, if we add and subtract
ED [g (x|D)] (26)
Remark: The expected output of the machine g (x|D)
75 / 96
Thus, we have that
Or Original variance
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)] + ED [g (x|D)]− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)])2 + ...
...2 ((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x]) + ...
... (ED [g (x|D)]− E [y|x])2)Finally
ED (((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x])) =? (27)
76 / 96
Thus, we have that
Or Original variance
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)] + ED [g (x|D)]− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)])2 + ...
...2 ((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x]) + ...
... (ED [g (x|D)]− E [y|x])2)Finally
ED (((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x])) =? (27)
76 / 96
Thus, we have that
Or Original variance
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)] + ED [g (x|D)]− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)])2 + ...
...2 ((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x]) + ...
... (ED [g (x|D)]− E [y|x])2)Finally
ED (((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x])) =? (27)
76 / 96
Thus, we have that
Or Original variance
VarD (g (x|D)) = ED((g (x|D)− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)] + ED [g (x|D)]− E [y|x])2)
= ED((g (x|D)− ED [g (x|D)])2 + ...
...2 ((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x]) + ...
... (ED [g (x|D)]− E [y|x])2)Finally
ED (((g (x|D)− ED [g (x|D)])) (ED [g (x|D)]− E [y|x])) =? (27)
76 / 96
We have the Bias-Variance
Our Final Equation
ED(
(g (x|D)− E [y|x])2) = ED(
(g (x|D)− ED [g (x|D)])2)︸ ︷︷ ︸VARIANCE
+ (ED [g (x|D)]− E [y|x])2︸ ︷︷ ︸BIAS
Where the varianceIt represent the measure of the error between our machine g (x|D) and theexpected output of the machine under xi ∼ p (x|Θ).
Where the biasIt represent the quadratic error between the expected output of themachine under xi ∼ p (x|Θ) and the expected output of the optimalregression.
77 / 96
We have the Bias-Variance
Our Final Equation
ED(
(g (x|D)− E [y|x])2) = ED(
(g (x|D)− ED [g (x|D)])2)︸ ︷︷ ︸VARIANCE
+ (ED [g (x|D)]− E [y|x])2︸ ︷︷ ︸BIAS
Where the varianceIt represent the measure of the error between our machine g (x|D) and theexpected output of the machine under xi ∼ p (x|Θ).
Where the biasIt represent the quadratic error between the expected output of themachine under xi ∼ p (x|Θ) and the expected output of the optimalregression.
77 / 96
We have the Bias-Variance
Our Final Equation
ED(
(g (x|D)− E [y|x])2) = ED(
(g (x|D)− ED [g (x|D)])2)︸ ︷︷ ︸VARIANCE
+ (ED [g (x|D)]− E [y|x])2︸ ︷︷ ︸BIAS
Where the varianceIt represent the measure of the error between our machine g (x|D) and theexpected output of the machine under xi ∼ p (x|Θ).
Where the biasIt represent the quadratic error between the expected output of themachine under xi ∼ p (x|Θ) and the expected output of the optimalregression.
77 / 96
We have the Bias-Variance
Our Final Equation
ED(
(g (x|D)− E [y|x])2) = ED(
(g (x|D)− ED [g (x|D)])2)︸ ︷︷ ︸VARIANCE
+ (ED [g (x|D)]− E [y|x])2︸ ︷︷ ︸BIAS
Where the varianceIt represent the measure of the error between our machine g (x|D) and theexpected output of the machine under xi ∼ p (x|Θ).
Where the biasIt represent the quadratic error between the expected output of themachine under xi ∼ p (x|Θ) and the expected output of the optimalregression.
77 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
78 / 96
Using this in our favor!!!
Something NotableIntroducing bias is equivalent to restricting the range of functions forwhich a model can account.Typically this is achieved by removing degrees of freedom.
ExamplesThey would be lowering the order of a polynomial or reducing the numberof weights in a neural network!!!
Ridge RegressionIt does not explicitly remove degrees of freedom but instead reduces theeffective number of parameters.
79 / 96
Using this in our favor!!!
Something NotableIntroducing bias is equivalent to restricting the range of functions forwhich a model can account.Typically this is achieved by removing degrees of freedom.
ExamplesThey would be lowering the order of a polynomial or reducing the numberof weights in a neural network!!!
Ridge RegressionIt does not explicitly remove degrees of freedom but instead reduces theeffective number of parameters.
79 / 96
Using this in our favor!!!
Something NotableIntroducing bias is equivalent to restricting the range of functions forwhich a model can account.Typically this is achieved by removing degrees of freedom.
ExamplesThey would be lowering the order of a polynomial or reducing the numberof weights in a neural network!!!
Ridge RegressionIt does not explicitly remove degrees of freedom but instead reduces theeffective number of parameters.
79 / 96
Using this in our favor!!!
Something NotableIntroducing bias is equivalent to restricting the range of functions forwhich a model can account.Typically this is achieved by removing degrees of freedom.
ExamplesThey would be lowering the order of a polynomial or reducing the numberof weights in a neural network!!!
Ridge RegressionIt does not explicitly remove degrees of freedom but instead reduces theeffective number of parameters.
79 / 96
Example
In the case of a linear regression model
C (w) =N∑
i=1
(di −wT xi
)2+ λ
d0∑j=1
w2j (28)
ThusThis is ridge regression (weight decay) and the regularizationparameter λ > 0 controls the balance between fitting the data andavoiding the penalty.A small value for λ means the data can be fit tightly without causinga large penalty.A large value for λ means a tight fit has to be sacrificed if it requireslarge weights.
80 / 96
Example
In the case of a linear regression model
C (w) =N∑
i=1
(di −wT xi
)2+ λ
d0∑j=1
w2j (28)
ThusThis is ridge regression (weight decay) and the regularizationparameter λ > 0 controls the balance between fitting the data andavoiding the penalty.A small value for λ means the data can be fit tightly without causinga large penalty.A large value for λ means a tight fit has to be sacrificed if it requireslarge weights.
80 / 96
Example
In the case of a linear regression model
C (w) =N∑
i=1
(di −wT xi
)2+ λ
d0∑j=1
w2j (28)
ThusThis is ridge regression (weight decay) and the regularizationparameter λ > 0 controls the balance between fitting the data andavoiding the penalty.A small value for λ means the data can be fit tightly without causinga large penalty.A large value for λ means a tight fit has to be sacrificed if it requireslarge weights.
80 / 96
Example
In the case of a linear regression model
C (w) =N∑
i=1
(di −wT xi
)2+ λ
d0∑j=1
w2j (28)
ThusThis is ridge regression (weight decay) and the regularizationparameter λ > 0 controls the balance between fitting the data andavoiding the penalty.A small value for λ means the data can be fit tightly without causinga large penalty.A large value for λ means a tight fit has to be sacrificed if it requireslarge weights.
80 / 96
Important
The BiasIt favors solutions involving small weights and the effect is to smooth theoutput function.
81 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
82 / 96
Now, we can carry out the optimization
First, we rewrite the cost function the following way
S (w) =N∑
i=1(di − f (xi))2 (29)
And we will use a generalized version for f
f (xi) =d1∑
j=1wjφj (xi) (30)
WhereThe free variables are the weights {wj}d1
j=1.
83 / 96
Now, we can carry out the optimization
First, we rewrite the cost function the following way
S (w) =N∑
i=1(di − f (xi))2 (29)
And we will use a generalized version for f
f (xi) =d1∑
j=1wjφj (xi) (30)
WhereThe free variables are the weights {wj}d1
j=1.
83 / 96
Now, we can carry out the optimization
First, we rewrite the cost function the following way
S (w) =N∑
i=1(di − f (xi))2 (29)
And we will use a generalized version for f
f (xi) =d1∑
j=1wjφj (xi) (30)
WhereThe free variables are the weights {wj}d1
j=1.
83 / 96
Where
φj (x i) is in our case, we may have the Gaussian distribution
φj (xi) = φ (xi ,xj) (31)
With
φ (x,xj) = exp{− 1
2σ2 ‖x − xi‖}
(32)
84 / 96
Where
φj (x i) is in our case, we may have the Gaussian distribution
φj (xi) = φ (xi ,xj) (31)
With
φ (x,xj) = exp{− 1
2σ2 ‖x − xi‖}
(32)
84 / 96
Thus
Final cost function assuming there is a regularization term per weight
C (w,λ) =N∑
i=1(di − f (xi))2 +
d1∑j=1
λjw2j (33)
What do we do?1 Differentiate the function with respect to the free variables.2 Equate the results with zero.3 Solve the resulting equations.
85 / 96
Thus
Final cost function assuming there is a regularization term per weight
C (w,λ) =N∑
i=1(di − f (xi))2 +
d1∑j=1
λjw2j (33)
What do we do?1 Differentiate the function with respect to the free variables.2 Equate the results with zero.3 Solve the resulting equations.
85 / 96
Thus
Final cost function assuming there is a regularization term per weight
C (w,λ) =N∑
i=1(di − f (xi))2 +
d1∑j=1
λjw2j (33)
What do we do?1 Differentiate the function with respect to the free variables.2 Equate the results with zero.3 Solve the resulting equations.
85 / 96
Thus
Final cost function assuming there is a regularization term per weight
C (w,λ) =N∑
i=1(di − f (xi))2 +
d1∑j=1
λjw2j (33)
What do we do?1 Differentiate the function with respect to the free variables.2 Equate the results with zero.3 Solve the resulting equations.
85 / 96
Differentiate the function with respect to the free variables.
First
∂C (w,λ)∂wj
= 2N∑
i=1(di − f (xi))
∂f (xi)∂wj
+ 2λjwj (34)
We get differential of ∂f (xi)∂wj
∂f (xi)∂wj
= φj (xi) (35)
86 / 96
Differentiate the function with respect to the free variables.
First
∂C (w,λ)∂wj
= 2N∑
i=1(di − f (xi))
∂f (xi)∂wj
+ 2λjwj (34)
We get differential of ∂f (xi)∂wj
∂f (xi)∂wj
= φj (xi) (35)
86 / 96
Now
We have thenN∑
i=1f (xi)φj (xi) + λjwj =
N∑i=1
diφj (xi) (36)
Something NotableThere are m such equations, for 1 ≤ j ≤ m, each representing oneconstraint on the solution.Since there are exactly as many constraints as there are unknownsequations has, except under certain pathological conditions, a uniquesolution.
87 / 96
Now
We have thenN∑
i=1f (xi)φj (xi) + λjwj =
N∑i=1
diφj (xi) (36)
Something NotableThere are m such equations, for 1 ≤ j ≤ m, each representing oneconstraint on the solution.Since there are exactly as many constraints as there are unknownsequations has, except under certain pathological conditions, a uniquesolution.
87 / 96
Now
We have thenN∑
i=1f (xi)φj (xi) + λjwj =
N∑i=1
diφj (xi) (36)
Something NotableThere are m such equations, for 1 ≤ j ≤ m, each representing oneconstraint on the solution.Since there are exactly as many constraints as there are unknownsequations has, except under certain pathological conditions, a uniquesolution.
87 / 96
Using Our Linear Algebra
We have then
φTj f + λjwj = φT
j d (37)
Where
φj =
φj (x1)φj (x2)
...φj (xN )
, f =
f (x1)f (x2)
...f (xN )
,d =
d1d2...
dN
(38)
88 / 96
Using Our Linear Algebra
We have then
φTj f + λjwj = φT
j d (37)
Where
φj =
φj (x1)φj (x2)
...φj (xN )
, f =
f (x1)f (x2)
...f (xN )
,d =
d1d2...
dN
(38)
88 / 96
NowSince there is one of these equations, each relating one scalarquantity to another, we can stack them
φT1 f
φT2 f...
φTd1f
+
λ1w1λ2w2...
λd1wd1
=
φT
1 dφT
2 d...
φTd1d
(39)
Now, if we define
Φ =[φ1 φ2 . . . φd1
](40)
Written in full form
Φ =
φ1 (x1) φ2 (x1) · · · φd1 (x1)φ1 (x2) φ2 (x2) · · · φd1 (x2)
...... . . . ...
φ1 (xN ) φ2 (xN ) · · · φd1 (xN )
(41)
89 / 96
NowSince there is one of these equations, each relating one scalarquantity to another, we can stack them
φT1 f
φT2 f...
φTd1f
+
λ1w1λ2w2...
λd1wd1
=
φT
1 dφT
2 d...
φTd1d
(39)
Now, if we define
Φ =[φ1 φ2 . . . φd1
](40)
Written in full form
Φ =
φ1 (x1) φ2 (x1) · · · φd1 (x1)φ1 (x2) φ2 (x2) · · · φd1 (x2)
...... . . . ...
φ1 (xN ) φ2 (xN ) · · · φd1 (xN )
(41)
89 / 96
NowSince there is one of these equations, each relating one scalarquantity to another, we can stack them
φT1 f
φT2 f...
φTd1f
+
λ1w1λ2w2...
λd1wd1
=
φT
1 dφT
2 d...
φTd1d
(39)
Now, if we define
Φ =[φ1 φ2 . . . φd1
](40)
Written in full form
Φ =
φ1 (x1) φ2 (x1) · · · φd1 (x1)φ1 (x2) φ2 (x2) · · · φd1 (x2)
...... . . . ...
φ1 (xN ) φ2 (xN ) · · · φd1 (xN )
(41)
89 / 96
We can then
Define the following matrix equation
ΦT f + Λw = ΦT d (42)
Where
Λ =
λ1 0 · · · 00 λ2 · · · 0...
... . . . ...0 0 · · · λd1
(43)
90 / 96
We can then
Define the following matrix equation
ΦT f + Λw = ΦT d (42)
Where
Λ =
λ1 0 · · · 00 λ2 · · · 0...
... . . . ...0 0 · · · λd1
(43)
90 / 96
Now, we have that
The vector can be decomposed into the product of two termsDesign matrix and the weight vector
We have then
fi = f (xi) =d1∑
j=1wjhj (xi) = φ
Ti w (44)
Where
φi =
φ1 (xi)φ2 (xi)
...φd1 (xi)
(45)
91 / 96
Now, we have that
The vector can be decomposed into the product of two termsDesign matrix and the weight vector
We have then
fi = f (xi) =d1∑
j=1wjhj (xi) = φ
Ti w (44)
Where
φi =
φ1 (xi)φ2 (xi)
...φd1 (xi)
(45)
91 / 96
Now, we have that
The vector can be decomposed into the product of two termsDesign matrix and the weight vector
We have then
fi = f (xi) =d1∑
j=1wjhj (xi) = φ
Ti w (44)
Where
φi =
φ1 (xi)φ2 (xi)
...φd1 (xi)
(45)
91 / 96
Furthermore
We get that
f =
f1f2...
fN
=
φ
T1 w
φT2 w...
φTN w
= Φw (46)
Finally, we have that
ΦT d =ΦT f + Λw=ΦT Φw + Λw
=[ΦT Φ + Λ
]w
92 / 96
Furthermore
We get that
f =
f1f2...
fN
=
φ
T1 w
φT2 w...
φTN w
= Φw (46)
Finally, we have that
ΦT d =ΦT f + Λw=ΦT Φw + Λw
=[ΦT Φ + Λ
]w
92 / 96
Now...
We get finally
w =[ΦT Φ + Λ
]−1ΦT d (47)
RememberThis equation is the most general form of the normal equation.
We have two casesIn standard ridge regression λj = λ, 1 ≤ j ≤ m.Ordinary least squares where there is no weight penalty or all λj = 0,1 ≤ j ≤ m..
93 / 96
Now...
We get finally
w =[ΦT Φ + Λ
]−1ΦT d (47)
RememberThis equation is the most general form of the normal equation.
We have two casesIn standard ridge regression λj = λ, 1 ≤ j ≤ m.Ordinary least squares where there is no weight penalty or all λj = 0,1 ≤ j ≤ m..
93 / 96
Now...
We get finally
w =[ΦT Φ + Λ
]−1ΦT d (47)
RememberThis equation is the most general form of the normal equation.
We have two casesIn standard ridge regression λj = λ, 1 ≤ j ≤ m.Ordinary least squares where there is no weight penalty or all λj = 0,1 ≤ j ≤ m..
93 / 96
Thus, we have
First Case
w =[ΦT Φ + λI d1
]−1ΦT d (48)
Second Case
w =[ΦT Φ
]−1ΦT d (49)
94 / 96
Thus, we have
First Case
w =[ΦT Φ + λI d1
]−1ΦT d (48)
Second Case
w =[ΦT Φ
]−1ΦT d (49)
94 / 96
Outline1 Introduction
Main IdeaBasic Radial-Basis Functions
2 SeparabilityCover’s Theorem on the separability of patternsDichotomyφ-separable functionsThe Stochastic ExperimentThe XOR ProblemSeparating Capacity of a Surface
3 Interpolation ProblemWhat is gained?Feedforward NetworkLearning ProcessRadial-Basis Functions (RBF)
4 IntroductionDescription of the ProblemWell-posed or ill-posedThe Main Problem
5 Regularization TheorySolving the issueBias-Variance DilemmaMeasuring the difference between optimal and learnedThe Bias-Variance
How can we use this?Getting a solutionWe still need to talk about...
95 / 96
There are still several things that we need to look at...
FirstWhat is the variance of the weight vector? The Variance Matrix.
SecondThe prediction of the output at any of the training set inputs - TheProjection Matrix
FinallyThe incremental algorithm for the problem!!!
96 / 96
There are still several things that we need to look at...
FirstWhat is the variance of the weight vector? The Variance Matrix.
SecondThe prediction of the output at any of the training set inputs - TheProjection Matrix
FinallyThe incremental algorithm for the problem!!!
96 / 96
There are still several things that we need to look at...
FirstWhat is the variance of the weight vector? The Variance Matrix.
SecondThe prediction of the output at any of the training set inputs - TheProjection Matrix
FinallyThe incremental algorithm for the problem!!!
96 / 96