Reproducing Kernel Banach Spaces for Machine...
Embed Size (px)
Transcript of Reproducing Kernel Banach Spaces for Machine...

Journal of Machine Learning Research 10 (2009) 27412775 Submitted 2/09; Revised 7/09; Published 12/09
Reproducing Kernel Banach Spaces for Machine Learning
Haizhang Zhang [email protected] Xu [email protected] of MathematicsSyracuse UniversitySyracuse, NY 13244, USA
Jun Zhang [email protected] of PsychologyUniversity of MichiganAnn Arbor, MI 48109, USA
Editor: Ingo Steinwart
AbstractWe introduce the notion of reproducing kernel Banach spaces (RKBS) and study special semiinnerproduct RKBS by making use of semiinnerproducts and the duality mapping. Properties ofan RKBS and its reproducing kernel are investigated. As applications, we develop in the frameworkof RKBS standard learning schemes including minimal norm interpolation, regularization network,support vector machines, and kernel principal component analysis. In particular, existence, uniqueness and representer theorems are established.Keywords: reproducing kernel Banach spaces, reproducing kernels, learning theory, semiinnerproducts, representer theorems
1. Introduction
Learning a function from its finite samples is a fundamental science problem. The essence in achieving this is to choose an appropriate measurement of similarities between elements in the domain ofthe function. A recent trend in machine learning is to use a positive definite kernel (Aronszajn, 1950)to measure the similarity between elements in an input space X (Scholkopf and Smola, 2002; ShaweTaylor and Cristianini, 2004; Vapnik, 1998; Xu and Zhang, 2007, 2009). Set Nn := {1,2, . . . ,n}for n N. A function K : X X C is called a positive definite kernel if for all finite subsetsx := {x j : j Nn} X the matrix
K[x] := [K(x j,xk) : j,k Nn] (1)
is hermitian and positive semidefinite. The reason of using positive definite kernels to measuresimilarity lies in the celebrated theoretical fact due to Mercer (1909) that there is a bijective correspondence between them and reproducing kernel Hilbert spaces (RKHS). An RKHS H on X is aHilbert space of functions on X for which point evaluations are always continuous linear functionals. One direction of the bijective correspondence says that if K is a positive definite kernel on Xthen there exists a unique RKHS H on X such that K(x, ) H for each x X and for all f Hand y X
f (y) = ( f ,K(y, ))H , (2)
c2009 Haizhang Zhang, Yuesheng Xu and Jun Zhang.

ZHANG, XU AND ZHANG
where (, )H denotes the inner product on H . Conversely, if H is an RKHS on X then there is aunique positive definite kernel K on X such that {K(x, ) : x X} H and (2) holds. In light of thisbijective correspondence, positive definite kernels are usually called reproducing kernels.
By taking f := K(x, ) for x X in Equation (2), we get that
K(x,y) = (K(x, ),K(y, ))H , x,y X . (3)
Thus K(x,y) is represented as an inner product on an RKHS. This explains why K(x,y) is able tomeasure similarities of x and y. The advantages brought by the use of an RKHS include: (1) theinputs can be handled and explained geometrically; (2) geometric objects such as hyperplanes areprovided by the RKHS for learning; (3) the powerful tool of functional analysis applies (Scholkopfand Smola, 2002). Based on the theory of reproducing kernels, many effective schemes have beendeveloped for learning from finite samples (Evgeniou et al., 2000; Micchelli et al., 2009; Scholkopfand Smola, 2002; ShaweTaylor and Cristianini, 2004; Vapnik, 1998). In particular, the widelyused regularized learning algorithm works by generating a predictor function from the training data{(x j,y j) : j Nn} X C as the minimizer of
minfHK
jNn
L( f (x j),y j)+ f2HK
, (4)
where HK denotes the RKHS corresponding to the positive definite kernel K, L is a prescribed lossfunction, and is a positive regularization parameter.
This paper is motivated from machine learning in Banach spaces. There are advantages oflearning in Banach spaces over Hilbert spaces. Firstly, there is essentially only one Hilbert spaceonce the dimension of the space is fixed. This follows from the wellknown fact that any two Hilbertspaces over C of the same dimension are isometrically isomorphic. By contrast, for p 6= q [1,+],Lp[0,1] and Lq[0,1] are not isomorphic, namely, there does not exist a bijective bounded linearmapping between them (see, Fabian et al., 2001, page 180). Thus, compared to Hilbert spaces,Banach spaces possess much richer geometric structures, which are potentially useful for developinglearning algorithms. Secondly, in some applications, a norm from a Banach space is invoked withoutbeing induced from an inner product. For instance, it is known that minimizing about the p normon Rd leads to sparsity of the minimizer when p is close to 1 (see, for example, Tropp, 2006). In theextreme case that : Rd [0,+) is strictly concave and > 0, one can show that the minimizerfor
min{(x)+x1 : x Rd} (5)
has at most one nonzero element. The reason is that the extreme points on a sphere in the 1 normmust lie on axes of the Euclidean coordinate system. A detailed proof of this result is providedin the appendix. Thirdly, since many training data come with intrinsic structures that make themimpossible to be embedded into a Hilbert space, learning algorithms based on RKHS may not workwell for them. Hence, there is a need to modify the algorithms by adopting norms in Banach spaces.For example, one might have to replace the norm HK in (4) with that of a Banach space.
There has been considerable work on learning in Banach spaces in the literature. ReferencesBennett and Bredensteiner (2000); Micchelli and Pontil (2004, 2007); Micchelli et al. (2003); Zhang(2002) considered the problem of minimizing a regularized functional of the form
jNn
L( j( f ),y j)+( fB), f B,
2742

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
where B is Banach space, j are in the dual B, y j C, L is a loss function, and is a strictlyincreasing nonnegative function. In particular, Micchelli et al. (2003) considered learning in Besovspaces (a special type of Banach spaces). Online learning in finite dimensional Banach spaces wasstudied, for example, in Gentile (2001). Learning of an Lp function was considered in Kimber andLong (1995). Classifications in Banach spaces, and more generally in metric spaces were discussedin Bennett and Bredensteiner (2000), Der and Lee (2007), Hein et al. (2005), von Luxburg andBousquet (2004) and Zhou et al. (2002).
The above discussion indicates that there is a need of introducing the notion of reproducingkernel Banach spaces for the systematic study of learning in Banach spaces. Such a definition isexpected to result in consequences similar to those in an RKHS. A generalization of RKHS to nonHilbert spaces using point evaluation with kernels was proposed in Canu et al. (2003), although thespaces considered there might be too general to have favorable properties of an RKHS. We shallintroduce the notion of reproducing kernel Banach spaces in Section 2, and a general constructionin Section 3. It will become clear that the lack of an inner product may cause arbitrariness in theproperties of the associated reproducing kernel. To overcome this, we shall establish in Section 4s.i.p. reproducing kernel Banach spaces by making use of semiinnerproducts for normed vectorspaces first defined by Lumer (1961) and further developed by Giles (1967). Semiinnerproductswere first applied to machine learning by Der and Lee (2007) to develop hard margin hyperplaneclassification in Banach spaces. Here the availability of a semiinnerproduct enables us to studybasic properties of reproducing kernel Banach spaces and their reproducing kernels. In Section 5,we shall develop in the framework of reproducing kernel Banach spaces standard learning schemesincluding minimal norm interpolation, regularization network, support vector machines, and kernelprincipal component analysis. Existence, uniqueness and representer theorems for the learningschemes will be proved. We draw conclusive remarks in Section 6 and include two technical resultsin Appendix.
2. Reproducing Kernel Banach Spaces
Without specifically mentioned, all vector spaces in this paper are assumed to be complex. Let Xbe a prescribed input space. A normed vector space B is called a Banach space of functions on Xif it is a Banach space whose elements are functions on X , and for each f B , its norm fB inB vanishes if and only if f , as a function, vanishes everywhere on X . By this definition, Lp[0,1],1 p +, is not a Banach space of functions as it consists of equivalent classes of functions withrespect to the Lebesgue measure.
Influenced by the definition of RKHS, our first intuition is to define a reproducing kernel Banachspace (RKBS) as a Banach space of functions on X on which point evaluations are continuous linearfunctionals. If such a definition was adopted then the first example that comes to our mind wouldbe C[0,1], the Banach space of continuous functions on [0,1] equipped with the maximum norm. Itsatisfies the definition. However, since for each f C[0,1],
f (x) = x( f ), x [0,1],
the reproducing kernel for C[0,1] would have to be the delta distribution, which is not a function thatcan be evaluated. This example suggests that there should exist a way of identifying the elementsin the dual of an RKBS with functions. Recall that two normed vector spaces V1 and V2 are saidto be isometric if there is a bijective linear normpreserving mapping between them. We call such
2743

ZHANG, XU AND ZHANG
V1 and V2 an identification of each other. We would like the dual space B of an RKBS B on Xto be isometric to a Banach space of functions on X . In addition to this requirement, later on wewill find it very convenient to jump freely between a Banach space and its dual. For this reason, wewould like an RKBS B to be reflexive in the sense that (B) = B . The above discussion leads tothe following formal definition.
Definition 1 A reproducing kernel Banach space (RKBS) on X is a reflexive Banach space B offunctions on X for which B is isometric to a Banach space B# of functions on X and the pointevaluation is continuous on both B and B#.
Several remarks are in order about this definition. First, whether B is an RKBS is independentof the choice of the identification B# of B. In other words, if the point evaluation is continuouson some identification B# then it is continuous on all the identifications. The reason is that any twoidentifications of B are isometric. Second, an RKHS H on X is an RKBS. To see this, we set
H # := { f : f H } (6)
with the norm fH # := fH , where f denotes the conjugate of f defined by f (x) := f (x), x X .By the Riesz representation theorem (Conway, 1990), each u H has the form
u( f ) = ( f , f0)H , f H
for some unique f0 H and uH = f0H . We introduce a mapping : H H # by setting
(u) := f0.
Clearly, so defined is isometric from H to H #. We conclude that an RKHS is a special RKBS.Third, the identification B# of B of an RKBS is usually not unique. However, since they areisometric to each other, we shall assume that one of them has been chosen for an RKBS B underdiscussion. In particular, the identification of H of an RKHS H will always be chosen as (6).Fourth, for notational simplicity, we shall still denote the fixed identification of B by B. Let uskeep in mind that originally B consists of continuous linear functionals on B . Thus, when we shallbe treating elements in B as functions on X , we actually think B as its chosen identification. Withthis notational convention, we state our last remark that if B is an RKBS on X then so is B.
We shall show that there indeed exists a reproducing kernel for an RKBS. To this end, weintroduce for a normed vector space V the following bilinear form on V V by setting
(u,v)V := v(u), u V, v V .
It is called bilinear for the reason that for all , C, u,v V , and u,v V there holds
(u+v,u)V = (u,u)V +(v,u)V
and(u,u +v)V = (u,u)V +(u,v)V .
Note that if V is a reflexive Banach space then for any continuous linear functional T on V thereexists a unique u V such that
T (v) = (u,v)V , v V .
2744

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Theorem 2 Suppose that B is an RKBS on X. Then there exists a unique function K : X X Csuch that the following statements hold.(a) For every x X, K(,x) B and
f (x) = ( f ,K(,x))B , for all f B.
(b) For every x X, K(x, ) B and
f (x) = (K(x, ), f )B , for all f B. (7)
(c) The linear span of {K(x, ) : x X} is dense in B , namely,
span{K(x, ) : x X} = B. (8)
(d) The linear span of {K(,x) : x X} is dense in B, namely,
span{K(,x) : x X} = B. (9)
(e) For all x,y XK(x,y) = (K(x, ),K(,y))B . (10)
Proof For every x X , since x is a continuous linear functional on B , there exists gx B suchthat
f (x) = ( f ,gx)B , f B.
We introduce a function K on X X by setting
K(x,y) := gx(y), x,y X .
It follows that K(x, ) B for each x X , and
f (x) = ( f , K(x, ))B , f B, x X . (11)
There is only one function on X X with the above properties. Assume to the contrary that there isanother G : X X C satisfying {G(x, ) : x X} B and
f (x) = ( f , G(x, ))B , f B, x X .
The above equation combined with (11) yields that
( f , K(x, ) G(x, ))B = 0, for all f B, x X .
Thus, K(x, ) G(x, ) = 0 in B for each x X . Since B is a Banach space of functions on X , weget for every y X that
K(x,y) G(x,y) = 0,
that is, K = G.Likewise, there exists a unique K : X X C such that K(y, ) B , y X and
f (y) = (K(y, ), f )B , f B, y X . (12)
2745

ZHANG, XU AND ZHANG
Letting f := K(y, ) in (11) yields that
K(y,x) = (K(y, ), K(x, ))B , x,y X , (13)
and setting f := K(x, ) in (12) ensures that
K(x,y) = (K(y, ), K(x, ))B , x,y X .
Combining the above equation with (13), we get that
K(x,y) = K(y,x), x,y X .
Therefore, K satisfies (a) and (b) as stated in the theorem. Equation (10) in (e) is proved by lettingf = K(,y) in (7). To complete the proof, we shall show (c) only, since (d) can be handled in asimilar way. Suppose that (8) does not hold. Then by the HahnBanach theorem, there exists anontrivial functional f B such that
(K(x, ), f )B = 0, for all x X .
We get immediately from (12) that f (x) = 0 for all x X . Since B is a Banach space of functionson X , f = 0 in B, a contradiction.
We call the function K in Theorem 2 the reproducing kernel for the RKBS B . By Theorem2, an RKBS has exactly one reproducing kernel. However, different RKBS may have the samereproducing kernel. Examples will be given in the next section. This results from a fundamentaldifference between Banach spaces and Hilbert spaces. To explain this, we let W be a Banach spaceand V a subset of W such that spanV is dense in W . Suppose that a norm on elements of V isprescribed. If W is a Hilbert space and an inner product is defined among elements in V , then thenorm extends in a unique way to spanV , and hence to the whole space W . Assume now that Wis only known to be a Banach space and V W satisfying spanV = W is given. Then even ifa bilinear form is defined between elements in V and those in V , the norm may not have a uniqueextension to the whole space W . Consequently, although we have at hand a reproducing kernel Kfor an RKBS B , the relationship (13), and denseness conditions (8), (9), we still can not determinethe norm on B .
3. Construction of Reproducing Kernels via Feature Maps
In this section, we shall characterize reproducing kernels for RKBS. The characterization will at thesame time provide a convenient way of constructing reproducing kernels and their correspondingRKBS. For the corresponding results in the RKHS case, see, for example, Saitoh (1997), Scholkopfand Smola (2002), ShaweTaylor and Cristianini (2004) and Vapnik (1998).
Theorem 3 Let W be a reflexive Banach space with dual space W . Suppose that there exists : X W , and : X W such that
span(X) =W , span(X) =W . (14)
2746

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Then B := {(u,())W : u W } with norm
(u,())W B := uW (15)
is an RKBS on X with the dual space B := {((),u)W : u W } endowed with the norm
((),u)W B := uW
and the bilinear form
((u,())W ,((),u)W )B := (u,u)W , u W , u W . (16)
Moreover, the reproducing kernel K for B is
K(x,y) := ((x),(y))W , x,y X . (17)
Proof We first show that B defined above is a Banach space of functions on X . To this end, we setu W and assume that
(u,(x))W = 0, for all x X . (18)
Then by the denseness condition (14), (u,u)W = 0 for all u W , implying that u = 0. Con
versely, if u = 0 in W then it is clear that (18) holds true. These arguments also show that therepresenter u W for a function (u,())W in B is unique. It is obvious that (15) defines anorm on B and B is complete under this norm. Therefore, B is a Banach space of functions on X .Similarly, so is B := {((),u)W : u W } equipped with the norm
((),u)W B := uW .
Define the bilinear form T on B B by setting
T ((u,())W ,((),u)W ) := (u,u)W , u W , u W .
Clearly, we have for all u W , u W that
T ((u,())W ,((),u)W ) uW uW = (u,())W B ((),u)W B .
Therefore, each function in B is a continuous linear functional on B . Note that the linear mappingu (u,())W is isometric from W to B . As a consequence, functions in B exhaust all thecontinuous linear functionals on B . We conclude that B = B with the bilinear form (16). Likewise,one can show that B is the dual of B by the reflexivity of W . We have hence proved that B isreflexive with dual B.
It remains to show that point evaluations are continuous on B and B. To this end, we get foreach x X and f := (u,())W , u W that
 f (x) = (u,(x))W  uW (x)W = fB(x)W ,
which implies that x is continuous on B . By similar arguments, it is continuous on B. Combiningall the discussion above, we reach the conclusion that B is an RKBS on X .
2747

ZHANG, XU AND ZHANG
For the function K on X X defined by (17), we get that K(x, ) B and K(,x) B for allx X . It is also verified that for f := (u,())W , u W
( f ,K(,x))B = ((u,())W ,((),(x))W )B = (u,(x))W = f (x).
Similarly, for f := ((),u)W , u W
(K(x, ), f )B = (((x),())W ,((),u)W )B = ((x),u)W = f (x).
These facts show that K is the reproducing kernel for B and complete the proof.
We call the mappings , in Theorem 3 a pair of feature maps for the reproducing kernel K.The spaces W , W are called the pair of feature spaces associated with the feature maps for K.As a corollary to Theorem 3, we obtain the following characterization of reproducing kernels forRKBS.
Theorem 4 A function K : X X C is the reproducing kernel of an RKBS on X if and only if itis of the form (17), where W is a reflexive Banach space, and mappings : X W , : X W satisfy (14).
Proof The sufficiency has been shown by the last theorem. For the necessity, we assume that K isthe reproducing kernel of an RKBS B on X , and set
W := B, W := B, (x) := K(x, ), (x) := K(,x), x X .
By Theorem 2, W ,W ,, satisfy all the conditions.
To demonstrate how we get RKBS and their reproducing kernels by Theorem 3, we now presenta nontrivial example of RKBS. Set X := R, I := [ 12 ,
12 ], and p (1,+). We make the convention
that q is always the conjugate number of p, that is, p1 +q1 = 1. Define W := Lp(I), W := Lq(I)and : X W , : X W as
(x)(t) := ei2xt , (x)(t) := ei2xt , x R, t I.
For f L1(R), its Fourier transform f is defined as
f (t) :=Z
R
f (x)ei2xtdx, t R,
and its inverse Fourier transform f is defined by
f (t) :=Z
R
f (x)ei2xtdx, t R.
The Fourier transform and the inverse Fourier transform can be defined on tempered distributions.Since the Fourier transform is injective on L1(R) (see, Rudin, 1987, page 185), the denseness requirement (14) is satisfied.
2748

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
By the construction described in Theorem 3, we obtain
B := { f C(R) : supp f I, f Lp(I)} (19)
with norm fB := fLp(I), and the dual
B := {g C(R) : supp g I, g Lq(I)}
with norm gB := gLq(I). For each f B and g B, we have
( f ,g)B =Z
I
f (t)g(t)dt.
The kernel K for B is given as
K(x,y) = ((x),(y))W =Z
I
ei2xtei2ytdt =sin(x y)
(x y)= sinc(x y).
We check that for each f B
( f ,K(,x))B =Z
I
f (t)(K(,x))(t)dt =Z
I
f (t)ei2xtdt = f (x), x R
and for each g B
(K(x, ),g)B =Z
I
(K(x, ))(t)g(t)dt =Z
I
g(t)ei2xtdt = g(x), x R.
When p = q = 2, B reduces to the classical space of bandlimited functions.In the above example, B is isometrically isomorphic to Lp(I). As mentioned in the introduction,
Lp(I) with different p are not isomorphic to each other. As a result, for different indices p the spacesB defined by (19) are essentially different. However, we see that they all have the sinc function asthe reproducing kernel. In fact, if no further conditions are imposed on an RKBS, its reproducingkernel can be rather arbitrary. We make a simple observation below to illustrate this.
Proposition 5 If the input space X is a finite set, then any nontrivial function K on X X is thereproducing kernel of some RKBS on X.
Proof Let K be an arbitrary nontrivial function on X X . Assume that X = Nm for some m N.Let d N be the rank of the matrix K[X ] as defined by (1). By elementary linear algebra, there existnonsingular matrices P,Q Cmm such that the transpose (K[X ])T of K[X ] has the form
(K[X ])T = P
[Id 00 0
]Q = P
[Id0
][Id 0
]Q, (20)
where Id is the d d identity matrix. For j Nm, let Pj be the transpose of the jth row of P[
Id0
]
and Q j the jth column of[
Id 0]
Q. Choose an arbitrary p (1,+). Equation (20) is rewrittenas
K( j,k) = (Q j,Pk)lp(Nd), j,k Nm. (21)
2749

ZHANG, XU AND ZHANG
We set W := lp(Nd), W := lq(Nd) and ( j) := Q j, ( j) := Pj, j Nm. Since P,Q are nonsingular, (14) holds true. Also, we have by (21) that
K( j,k) = (( j),(k))W , j,k Nm.
By Theorem 4, K is a reproducing kernel for some RKBS on X .
Proposition 5 reveals that due to the lack of an inner product, the reproducing kernel for a generalRKBS can be an arbitrary function on X X . Particularly, it might be nonsymmetric or nonpositivedefinite. In order for reproducing kernels of RKBS to have desired properties as those of RKHS, wemay need to impose certain structures on RKBS, which in some sense are substitutes of the innerproduct for RKHS. For this purpose, we shall adopt the semiinnerproduct introduced by Lumer(1961). A semiinnerproduct possesses some but not all properties of an inner product. Hilbertspace type arguments and results become available with the presence of a semiinnerproduct. Weshall introduce the notion of semiinnerproduct RKBS.
4. S.i.p. Reproducing Kernel Banach Spaces
The purpose of this section is to establish the notion of semiinnerproduct RKBS and study its properties. We start with necessary preliminaries on semiinnerproducts (Giles, 1967; Lumer, 1961).
4.1 SemiInnerProducts
A semiinnerproduct on a vector space V is a function, denoted by [, ]V , from V V to C suchthat for all x,y,z V and C
1. [x+ y,z]V = [x,z]V +[y,z]V ,
2. [x,y]V = [x,y]V , [x,y]V = [x,y]V ,
3. [x,x]V > 0 for x 6= 0,
4. (CauchySchwartz) [x,y]V 2 [x,x]V [y,y]V .
The property that [x,y]V = [x,y]V was not required in the original definition by Lumer (1961).We include it here for the observation by Giles (1967) that this property can always be imposed.
It is necessary to point out the difference between a semiinnerproduct and an inner product. Ingeneral, a semiinnerproduct [, ]V does not satisfy the conjugate symmetry [x,y]V = [y,x]V for allx,y V . As a consequence, there always exist x,y,z V such that
[x,y+ z]V 6= [x,y]V +[x,z]V .
In fact, a semiinnerproduct is always additive about the second variable only if it degenerates toan inner product. We show this fact below.
Proposition 6 A semiinnerproduct [, ]V on a complex vector space V is an inner product if andonly if
[x,y+ z]V = [x,y]V +[x,z]V , for all x,y,z V. (22)
2750

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Proof Suppose that V has a semiinnerproduct [, ]V that satisfies (22). It suffices to show that forall x,y V ,
[x,y]V = [y,x]V . (23)
Set C. By the linearity on the first and the additivity on the second variable, we get that
[x+y,x+y]V = [x,x]V +[y,y]V +[y,x]V + [x,y]V .
Since [z,z]V 0 for all z V , we must have
[y,x]V + [x,y]V R.
Choosing = 1 yields that Im [y,x]V = Im [x,y]V . And the choice = i results that Re [y,x]V =Re [x,y]V . Therefore, (23) holds, which implies that [, ]V is an inner product on V .
It was shown in Lumer (1961) that a vector space V with a semiinnerproduct is a normed spaceequipped with
xV := [x,x]1/2V , x V. (24)
Therefore, if a vector space V has a semiinnerproduct, we always assume that its norm is inducedby (24) and call V an s.i.p. space. Conversely, every normed vector space V has a semiinnerproduct that induces its norm by (24) (Giles, 1967; Lumer, 1961). By the CauchySchwartz inequality, if V is an s.i.p. space then for each x V , y [y,x]V is a continuous linear functional onV . We denote this linear functional by x. Following this definition, we have that
[x,y]V = y(x) = (x,y)V , x,y V. (25)
In general, a semiinnerproduct for a normed vector space may not be unique. However, adifferentiation property of the norm will ensure the uniqueness. We call a normed vector space VGateaux differentiable if for all x,y V \{0}
limtR, t0
x+ tyV xVt
exists. It is called uniformly Frechet differentiable if the limit is approached uniformly on S(V )S(V ). Here, S(V ) := {u V : uV = 1} is the unit sphere of V . The following result is due to Giles(1967).
Lemma 7 If an s.i.p. space V is Gateaux differentiable then for all x,y V with x 6= 0
limtR, t0
x+ tyV xVt
=Re [y,x]xV
. (26)
The above lemma indicates that a Gateaux differentiable normed vector space has a uniquesemiinnerproduct. In fact, we have by (26) that
[x,y]V = yV
(lim
tR, t0
y+ txV yVt
+ i limtR, t0
iy+ txV yVt
), x,y V \{0}. (27)
2751

ZHANG, XU AND ZHANG
For this reason, if V is a Gateaux differentiable normed vector space we always assume that it isan s.i.p. space with the semiinnerproduct defined as above. Interested readers are referred to theappendix for a proof that (27) indeed defines an s.i.p.
We shall impose one more condition on an s.i.p. space that will lead to a Riesz representationtheorem. A normed vector space V is uniformly convex if for all > 0 there exists a > 0 suchthat
x+ yV 2 for all x,y S(V ) with x yV .
The space Lp(,), 1 < p < +, on a measure space (,F ,) is uniformly convex. In particular,by the parallelogram law, any inner product space is uniformly convex. By a remark in Conway(1990), page 134, a uniformly convex Banach space is reflexive. There is a wellknown relationshipbetween uniform Frechet differentiability and uniform convexity (Cudia, 1963). It states that anormed vector space is uniformly Frechet differentiable if and only if its dual is uniformly convex.Therefore, if B is a uniformly convex and uniformly Frechet differentiable Banach space then sois B since B is reflexive. The important role of uniform convexity is displayed in the next lemma(Giles, 1967).
Lemma 8 (Riesz Representation Theorem) Suppose thatB is a uniformly convex, uniformly Frechetdifferentiable Banach space. Then for each f B there exists a unique x B such that f = x,that is,
f (y) = [y,x]B , y B.
Moreover, fB = xB .
The above Riesz representation theorem is desirable for RKBS. By Lemma 8 and the discussionright before it, we shall investigate in the next subsection RKBS which are both uniformly convexand uniformly Frechet differentiable.
Let B be a uniformly convex and uniformly Frechet differentiable Banach space. By Lemma 8,x x defines a bijection from B to B that preserves the norm. Note that this duality mapping isin general nonlinear. We call x the dual element of x. Since B is uniformly Frechet differentiable,it has a unique semiinnerproduct, which is given by
[x,y]B = [y,x]B , x,y B. (28)
We close this subsection with a concrete example of uniformly convex and uniformly Frechetdifferentiable Banach spaces. Let (,F ,) be a measure space and B := Lp(,) for some p (1,+). It is uniformly convex and uniformly Frechet differentiable with dual B = Lq(,). Foreach f B , its dual element in B is
f =f  f p2
fp2Lp(,). (29)
Consequently, the semiinnerproduct on B is
[ f ,g]B = g( f ) =
R
f ggp2d
gp2Lp(,).
With the above preparation, we shall study a special kind of RKBS which have desired properties.
2752

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
4.2 S.i.p. RKBS
Let X be a prescribed input space. We call a uniformly convex and uniformly Frechet differentiableRKBS on X an s.i.p. reproducing kernel Banach space (s.i.p. RKBS). Again, we see immediately that an RKHS is an s.i.p. RKBS. Also, the dual of an s.i.p. RKBS remains an s.i.p. RKBS. Ans.i.p. RKBS B is by definition uniformly Frechet differentiable. Therefore, it has a unique semiinnerproduct, which by Lemma 8 represents all the interaction between B and B. This leads toa more specific representation of the reproducing kernel. Precisely, we have the following consequences.
Theorem 9 Let B be an s.i.p. RKBS on X and K its reproducing kernel. Then there exists a uniquefunction G : X X C such that {G(x, ) : x X} B and
f (x) = [ f ,G(x, )]B , for all f B, x X . (30)
Moreover, there holds the relationship
K(,x) = (G(x, )), x X (31)
andf (x) = [K(x, ), f ]B , for all f B, x X . (32)
Proof By Lemma 8, for each x X there exists a function Gx B such that f (x) = [ f ,Gx]B forall f B . We define G : X X C by G(x,y) := Gx(y), x,y X . We see that G(x, ) = Gx B ,x X , and there holds (30). By the uniqueness in the Riesz representation theorem, such a functionG is unique. To prove the remaining claims, we recall from Theorem 2 that the reproducing kernelK satisfies for each f B that
f (x) = ( f ,K(,x))B , x X . (33)
andf (x) = (K(x, ), f )B , x X . (34)
By (25), (30) and (33), we have for each x X that
( f ,(G(x, )))B = [ f ,G(x, )]B = f (x) = ( f ,K(,x))B , f B.
The above equation implies (31). Equation (25) also implies that
(K(x, ), f )B = [K(x, ), f ]B .
This together with equation (34) proves (32) and completes the proof.
We call the unique function G in Theorem 9 the s.i.p. kernel of the s.i.p. RKBS B . It coincideswith the reproducing kernel K when B is an RKHS. In general, when G = K in Theorem 9, wecall G an s.i.p. reproducing kernel. By (30), an s.i.p. reproducing kernel G satisfies the followinggeneralization of (3)
G(x,y) = [G(x, ),G(y, )]B , x,y X . (35)
We shall give a characterization of an s.i.p. reproducing kernel in terms of its corresponding featuremap. To this end, for a mapping from X to a uniformly convex and uniformly Frechet differentiable Banach space W , we denote by the mapping from X to W defined as
(x) := ((x)), x X .
2753

ZHANG, XU AND ZHANG
Theorem 10 Let W be a uniformly convex and uniformly Frechet differentiable Banach space and a mapping from X to W such that
span(X) =W , span(X) =W . (36)
Then B := {[u,()]W : u W } equipped with[[u,()]W , [v,()]W
]
B
:= [u,v]W (37)
and B := {[(),u]W : u W } with[[(),u]W , [(),v]W
]
B:= [v,u]W
are uniformly convex and uniformly Frechet differentiable Banach spaces. And B is the dual of Bwith the bilinear form
([u,()]W , [(),v]W
)
B
:= [u,v]W , u,v W . (38)
Moreover, the s.i.p. kernel G of B is given by
G(x,y) = [(x),(y)]W , x,y X , (39)
which coincides with its reproducing kernel K.
Proof We shall show (39) only. The other results can be proved using arguments similar to thosein Theorem 3 and those in the proof of Theorem 7 in Giles (1967). Let f B . Then there exists aunique u W such that f = [u,()]W . By (38), for y X ,
f (y) = [u,(y)]W = ([u,()]W , [(),(y)]W )B = ( f , [(),(y)]W )B .
Comparing the above equation with (33), we obtain that
K(,y) = [(),(y)]W . (40)
On the other hand, by (37), for x X
f (x) = [u,(x)]W = [[u,()]W , [(x),()]W ]B ,
which implies that the s.i.p. kernel of B is
G(x, ) = [(x),()]W . (41)
By (40) and (41),K(x,y) = G(x,y) = [(x),(y)]W ,
which completes the proof.
As a direct consequence of the above theorem, we have the following characterization of s.i.p.reproducing kernels.
2754

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Theorem 11 A function G on X X is an s.i.p. reproducing kernel if and only if it is of the form(39), where is a mapping from X to a uniformly convex and uniformly Frechet differentiableBanach space W satisfying (36).
Proof The sufficiency is implied by Theorem 10. For the necessity, suppose that G is an s.i.p.reproducing kernel for some s.i.p. RKBS B on X . We choose W = B and (x) := G(x, ). ThenG has the form (39) by equation (35). Moreover, by (8), span(X) is dense in W . Assume thatspan(X) is not dense in W . Then by the HahnBanach theorem and Lemma 8, there exists anontrivial f B such that [(x), f ]B = 0, x X . Thus, by (28) we get that
f (x) = [ f ,G(x, )]B = [ f ,(x)]W = [(x), f ]B = 0, x X .
We end up with a zero function f , a contradiction. The proof is complete.
The mapping and space W in the above theorem will be called a feature map and featurespace of the s.i.p. reproducing kernel G, respectively.
By the duality relation (31) and the denseness condition (9), the s.i.p kernel G of an s.i.p. RKBSB on X satisfies
span{(G(x, )) : x X} = B. (42)
It is also of the form (35). By Theorem 11, G is identical with the reproducing kernel K for B if andonly if
span{G(x, ) : x X} = B. (43)
If B is not a Hilbert space then the duality mapping from B to B is nonlinear. Thus, it may notpreserve the denseness of a linear span. As a result, (43) would not follow automatically from(42). Here we remark that for most finite dimensional s.i.p. RKBS, (42) implies (43). This is dueto the wellknown fact that for all n N, the set of n n singular matrices has Lebesgue measurezero in Cnn. Therefore, the s.i.p. kernel for most finite dimensional s.i.p. RKBS is the same as thereproducing kernel. Nevertheless, we shall give an explicit example to illustrate that the two kernelsmight be different.
For each p (1,+) and n N, we denote by p(Nn) the Banach space of vectors in Cn withnorm
ap(Nn) :=
(
jNn
a jp)1/p
, a = (a j : j Nn) Cn.
As pointed out at the end of Section 4.1, p(Nn) is uniformly convex and uniformly Frechet differentiable. Its dual space is q(Nn). To construct the example, we introduce three vectors in 3(N3)by setting
e1 := (2,9,1), e2 := (1,8,0), e3 := (5,5,3).
By (29), their dual elements in 32 (N3) are
e1 =1
(738)1/3(4,81,1), e2 =
1
(513)1/3(1,64,0), e3 =
1
(277)1/3(25,25,9).
It can be directly verified that {e1,e2,e3} is linearly independent but {e1,e2,e
3} is not. Therefore,
span{e1,e2,e3} = 3(N3) (44)
2755

ZHANG, XU AND ZHANG
whilespan{e1,e
2,e
3} $
32 (N3). (45)
With the above preparations, we let N3 be the input space, the function from N3 to 3(N3)defined by ( j) = e j, j N3, and B the space of all the functions u := [(),u]3(N3), u
3(N3),on N3. By equation (44),
u := u3(N3), u 3(N3)
defines a norm on B . It is clear that point evaluations are continuous on B under this norm. Furthermore, since the linear mapping u u is isometrically isomorphic from B to
32 (N3), B is a
uniformly convex and uniformly Frechet differentiable Banach space. By this fact, we obtain thatB is an s.i.p. RKBS with semiinnerproduct
[u,v]B = [v,u]3(N3), u,v 3(N3). (46)
The above equation implies that the s.i.p. kernel G for B is
G( j,k) = [ek,e j]3(N3), j,k N3. (47)
Recall that the reproducing kernel K for B satisfies the denseness condition (8). Consequently, toshow that G 6= K, it suffices to show that
span{G( j, ) : j N3} $ B. (48)
To this end, we notice by (45) that there exists a nonzero element v 3(N3) such that
[v,e j]3(N3) = (v,ej)3(N3) = 0, j N3.
As a result, the nonzero function v satisfies by (46) and (47) that
[G( j, ),v]B = [e j ,v]B = [v,e j]3(N3) = 0, j N3,
which proves (48), and implies that the s.i.p. kernel and reproducing kernel for B are different. By(45), this is essentially due to the reason that the second condition of (36) is not satisfied.
4.3 Properties of S.i.p. Reproducing Kernels
The existence of a semiinnerproduct makes it possible to study properties of RKBS and their reproducing kernels. For illustration, we present below three of these properties.
4.3.1 NONPOSITIVE DEFINITENESS
An n n matrix M over a number field F (C or R) is said to be positive semidefinite if for all(c j : j Nn) Fn
jNn
kNn
c jckM jk 0.
We shall consider positive semidefiniteness of matrices G[x] as defined in (1) for an s.i.p. reproducingkernel G on X .
2756

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Let : X W be a feature map for G, that is, (39) and (36) hold. By properties 3 and 4 in thedefinition of a semiinnerproduct, we have that
G(x,x) 0, x X (49)
andG(x,y)2 G(x,x)G(y,y), x,y X . (50)
Notice that if a complex matrix is positive semidefinite then it must be hermitian. Since a semiinnerproduct is in general not an inner product, we can not expect a complex s.i.p. kernel to bepositive definite. In the real case, inequalities (49) and (50) imply that G[x] is positive semidefinitefor all x X with cardinality less than or equal to two. However, G[x] might stop being positivesemidefinite if x contains more than two points. We shall give an explicit example to explain thisphenomenon.
Set p (1,+) and W := p(N2). We let X := R+ := [0,+) and (x) = (1,x), x X . Thus,
(x) =(1,xp1)
(1+ xp)p2
p
, x X .
Clearly, satisfies the denseness condition (36). The corresponding s.i.p. reproducing kernel G isconstructed as
G(x,y) = [(x),(y)]W =1+ xyp1
(1+ yp)p2
p
, x,y X . (51)
Proposition 12 For the s.i.p. reproducing kernel G defined by (51), matrix G[x] is positive semidefinite for all x = {x,y,z} X if and only if p = 2.
Proof If p = 2 then W is a Hilbert space. As a result, G is a positive definite kernel. Hence, for allfinite subsets x X , G[x] is positive semidefinite.
Assume that G[x] is positive semidefinite for all x = {x,y,z} X . Choose x := {0,1, t} wheret is a positive number to be specified later. Then we have by (51) that
G[x] =
1 22/p11
(1+ t p)12/p
1 22/p1+ t p1
(1+ t p)12/p
11+ t
212/p1+ t p
(1+ t p)12/p
.
Let M be the symmetrization of G[x] given as
M =
112
+22/p212
+1
2(1+ t p)12/p12
+22/p2 22/p1+ t
222/p+
1+ t p1
2(1+ t p)12/p12
+1
2(1+ t p)12/p1+ t
222/p+
1+ t p1
2(1+ t p)12/p1+ t p
(1+ t p)12/p
.
2757

ZHANG, XU AND ZHANG
Matrix M preserves the positive semidefiniteness of G[x]. Therefore, its determinant M must benonnegative. Through an analysis of the asymptotic behavior of the component of M as t goes toinfinity, we obtain that
M = t2
8
(2
2p 2
)2+(t), t > 0,
where is a function satisfying that
limt
(t)t2
= 0.
Therefore, M being always nonnegative forces 22p 2 = 0, which occurs only if p = 2.
By Proposition 12, nonpositive semidefiniteness is a characteristic of s.i.p. reproducing kernelsfor RKBS that distincts them from reproducing kernels for RKHS.
4.3.2 POINTWISE CONVERGENCE
If fn converges to f in an s.i.p. RKBS with its s.i.p. kernel G then fn(x) converges to f (x) for anyx X and the limit is uniform on the set where G(x,x) is bounded. This follows from (30) and theCauchySchwartz inequality by
 fn(x) f (x) = [ fn f ,G(x, )]B  fn fB
[G(x, ),G(x, )]B =
G(x,x) fn fB .
4.3.3 WEAK UNIVERSALITY
Suppose that X is metric space and G is an s.i.p. reproducing kernel on X . We say that G is universal if G is continuous on X X and for all compact subsets Z X , span{G(x, ) : x Z} isdense in C(Z) (Micchelli et al., 2006; Steinwart, 2001). Universality of a kernel ensures that it canapproximate any continuous target function uniformly on compact subsets of the input space. Thisis crucial for the consistence of the learning algorithms with the kernel. We shall discuss the casewhen X is itself a compact metric space. Here we are concerned with the ability of G to approximate any continuous target function on X uniformly. For this purpose, we call a continuous kernelG on a compact metric space X weakly universal if span{G(x, ) : x X} is dense in C(X). Weshall present a characterization of weak universality. The results in the cases of positive definitekernels and vectorvalued positive definite kernels have been established respectively in Micchelliet al. (2006) and Caponnetto et al. (2008).
Proposition 13 Let be a feature map from a compact metric space X to W such that both :X W and : X W are continuous. Then the s.i.p. reproducing kernel G defined by (39) iscontinuous on X X, and there holds in C(X) the equality of subspaces
span{G(x, ) : x X} = span{[u,()]W : u W }.
Consequently, G is weakly universal if and only if
span{[u,()]W : u W } = C(X).
2758

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Proof First, we notice by
G(x,y) = [(x),(y)]W = ((x),(y))W , x,y X
that G is continuous on X X . Similarly, for each u W , [u,()]W = (u,())W C(X). Nowsince
G(x, ) = [(x),()]W {[u,()]W : u W },
we have the inclusion
span{G(x, ) : x X} span{[u,()]W : u W }.
On the other hand, let uW . By denseness condition (36), there exists a sequence vn span{(x) :x X} that converges to u. Since G is continuous on the compact space X X , it is bounded.Thus, by the property of pointwise convergence discussed before, [vn,()]W converges in C(X) to[u,()]W . Noting that
[vn,()]W span{G(x, ) : x X}, n N,
we have the reverse inclusion
span{[u,()]W : u W } span{G(x, ) : x X},
which proves the result.
We remark that in the case that W is a Hilbert space, the idea in the above proof can be appliedto show with less effort the main result in Caponnetto et al. (2008) and Micchelli et al. (2006), thatis, for each compact subset Z X
span{G(x, ) : x Z} = span{[u,()]W : u W },
where the two closures are taken in C(Z). A key element missing in the Banach space is theorthogonal decomposition in a Hilbert space W :
W = (span(Z))(Z), Z X .
For a normed vector space V , we denote for each A V by A := {v V : (a,v)V = 0, a A},and for each B V , B := {a V : (a,v)V = 0, v B}. A Banach space W is in general notthe direct sum of span(Z) and (Z). In fact, closed subspaces in W may not always have analgebraic complement unless W is isomorphic to a Hilbert space (see, Conway, 1990, page 94).
Universality and other properties of s.i.p. reproducing kernels will be treated specially in futurework. One of the main purposes of this study is to apply the tool of s.i.p. reproducing kernels tolearning in Banach spaces. To be precise, we shall develop in the framework of s.i.p. RKBS severalstandard learning schemes.
2759

ZHANG, XU AND ZHANG
5. Representer Theorems for Standard Learning Schemes
In this section, we assume that B is an s.i.p. RKBS on X with the s.i.p. reproducing kernel G definedby a feature map : X W as in (39). We shall develop in this framework several standard learningschemes including minimal norm interpolation, regularization network, support vector machines,and kernel principal component analysis. For introduction and discussions of these widely usedalgorithms in RKHS, see, for example, Cucker and Smale (2002), Evgeniou et al. (2000), Micchelliand Pontil (2005), Scholkopf and Smola (2002), ShaweTaylor and Cristianini (2004) and Vapnik(1998).
5.1 Minimal Norm Interpolation
The minimal norm interpolation is to find, among all functions in B that interpolate a prescribed setof points, a function with the minimal norm. Let x := {x j : j Nn} be a fixed finite set of distinctpoints in X and set for each y := (y j : j Nn) Cn
Iy := { f B : f (x j) = y j, j Nn}.
Our purpose is to find f0 Iy such that
f0B = inf{ fB : f Iy} (52)
provided that Iy is nonempty. Our first concern is of course the condition ensuring that Iy isnonempty. To address this issue, let us recall the useful property of the s.i.p. reproducing kernelG:
f (x) = [ f ,G(x, )]B = [G(,x), f]B , x X , f B. (53)
Lemma 14 The set Iy is nonempty for any y Cn if and only if Gx := {G(,x j) : j Nn} is linearlyindependent in B.
Proof Observe that Iy is nonempty for any y Cn if and only if span{( f (x j) : j Nn) : f B} isdense in Cn. Using the reproducing property (53), we have for each (c j : j Nn) Cn that
jNn
c j f (x j) = jNn
c j[ f ,G(x j, )]B =
[
jNn
c jG(,x j), f
]
B
.
Thus,
jNn
c j f (x j) = 0, for all f B
if and only if
jNn
c jG(,x j) = 0.
This implies that Gx is linearly independent in B if and only if span{( f (x j) : j Nn) : f B} isdense in Cn. Therefore, Iy is nonempty for any y Cn if and only if Gx is linearly independent.
We next show that the minimal norm interpolation problem in B always has a unique solutionunder the hypothesis that Gx is linearly independent. The following useful property of a uniformlyconvex Banach space is crucial (see, for example, Istratescu, 1984, page 53).
2760

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Lemma 15 If V is a uniformly convex Banach space, then for any nonempty closed convex subsetA V and any x V there exists a unique x0 A such that
x x0V = inf{xaV : a A}.
Proposition 16 If Gx is linearly independent in B, then for any yCn there exists a unique f0 Iysatisfying (52).
Proof By Lemma 14, Iy is nonempty. Note also that it is closed and convex. Since B is uniformlyconvex, by Lemma 15, there exists a unique f0 Iy such that
f0B = 0 f0B = inf{0 fB = fB : f Iy}.
The above equation proves the result.
We shall establish a representation of the minimal norm interpolator f0. For this purpose, asimple observation is made based on the following fact connecting orthogonality with best approximation in s.i.p. spaces (Giles, 1967).
Lemma 17 If V is a uniformly Frechet differentiable normed vector space, then x+yV xVfor all C if and only if [y,x]V = 0.
Lemma 18 If Iy is nonempty then f0 Iy is the minimizer of (52) if and only if
[g, f0]B = 0, for all g I0. (54)
Proof Let f0 Iy. It is obvious that f0 is the minimizer of (52) if and only if
f0 +gB f0B , g I0.
Since I0 is a linear subspace of B , the result follows immediately from Lemma 17.
The following result is of the representer theorem type. For the representer theorem in learningwith positive definite kernels, see, for example, Argyriou et al. (2008), Kimeldorf and Wahba (1971)and Scholkopf et al. (2001).
Theorem 19 (Representer Theorem) Suppose that Gx is linearly independent in B and f0 is thesolution of the minimal norm interpolation (52). Then there exists c = (c j : j Nn) Cn such that
f 0 = jNn
c jG(,x j). (55)
Moreover, a function of the form in the right hand side above is the solution if and only if c satisfies[
G(,xk), jNn
c jG(,x j)
]
B= yk, k Nn. (56)
2761

ZHANG, XU AND ZHANG
Proof Note that (54) is equivalent to f 0 I0 and that I0 = Gx. Therefore, f0 satisfies (54) if and
only iff 0 (
Gx).
Recall a consequence of the HahnBanach theorem that in a reflexive Banach space B , for eachB B,
(B) = spanB. (57)
By this fact, (55) holds true for some c Cn.Suppose that f B is of the form f = jNn c jG(,x j) where c satisfies (56). Then f
spanGx. By (57), f satisfies (54). Furthermore, (56) implies that
f (xk) = [ f ,G(xk, )]B = [G(,xk), f]B =
[G(,xk),
jNn
c jG(,x j)
]
B= yk, k Nn. (58)
That is, f Iy. By Lemma 18, f = f0. On the other hand, f0 has the form (55). As shown in (58),(56) is simply the interpolation condition that f0(xk) = yk, k Nn. Thus, it must be true. The proofis complete.
Applying the inverse of the duality mapping to both sides of (55) yields a representation off0 in the space B . However, since the duality mapping is nonadditive unless B is an RKHS, thisprocedure in general does not result in a linear representation.
We conclude that under the condition that Gx is linearly independent, the minimal norm interpolation problem (52) has a unique solution, and finding the solution reduces to solving the system(56) of equations about c Cn. The solution c of (56) is unique by Theorem 19. Again, the difference from the result for RKHS is that (56) is often nonlinear about c since by Proposition 6 asemiinnerproduct is generally nonadditive about the second variable.
To see an explicit form of (56), we shall reformulate it in terms of the feature map from X toW . Let B and B be identified as in Theorem 10. Then (56) has the equivalent form
[(xk),
jNn
c j(x j)]
B= yk, k Nn.
In the particular case that W = Lp(,), p (1,+) on some measure space (,F ,), and W =Lq(,), the above equation is rewritten as
Z
(xk)
jNn
c j(x j)
jNn
c j(x j)q2
d = yk
jNn
c j(x j)
q2
Lq(,), k Nn.
5.2 Regularization Network
We consider learning a predictor function from a finite sample data z := {(x j,y j) : j Nn} X Cin this subsection. The predictor function will yield from a regularized learning algorithm. LetL : CC R+ be a loss function that is continuous and convex with respect to its first variable.For each f B , we set
Ez( f ) := jNn
L( f (x j),y j) and Ez,( f ) := Ez( f )+ f2B ,
2762

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
where is a positive regularization parameter. The predictor function that we learn from the sampledata z will be the function f0 satisfying
Ez,( f0) = inf{Ez,( f ) : f B}. (59)
One can always make this choice as we shall prove that the minimizer of (59) exists and is unique.
Theorem 20 There exists a unique f0 B satisfying (59).
Proof We first show the existence. If f B satisfies f2B
> 1Ez,(0) then
Ez,( f ) f2B > Ez,(0).
Thus
inf{Ez,( f ) : f B} = inf
{Ez,( f ) : f B, f
2B
1Ez,(0)
}.
Let
e := inf
{Ez,( f ) : f B, f
2B
1Ez,(0)
}
and
A :=
{f B : f2B
1Ez,(0)
}.
Then, there exists a sequence fk A, k N, such that
e Ez,( fk) e+1k. (60)
Since B is reflexive, A is weakly compact, that is, we may assume that there exists f0 A such thatfor all g B
limk
[ fk,g]B = [ f0,g]B . (61)
In particular, the choice g := G(x j, ), j Nn yields that fk(x j) converges to f0(x j) as k for allj Nn. Since the loss function L is continuous about the first variable, we have that
limk
Ez( fk) = Ez( f0).
Also, choosing g = f0 in (61) yields that
limk
[ fk, f0]B = f02B .
By the above two equations, for each > 0 there exists some N such that for k N
Ez( f0) Ez( fk)+
and f0
2B f0
2B + [ fk, f0]B  f0
2B + fkB f0B , if f0 6= 0.
If f0 = 0, then trivially we have
f0B f0B + fkB .
2763

ZHANG, XU AND ZHANG
Combining the above three equations, we get for k N that
Ez,( f0) 1
(1 )2Ez,( fk)+ .
By (60) and the arbitrariness of , we conclude that f0 is a minimizer of (59).Since L is convex with respect to its first variable and 2
Bis strictly convex, Ez, is strictly
convex on B . This implies the uniqueness of the minimizer.
In the rest of this subsection, we shall consider the regularization network in B , that is, the lossfunction L is specified as
L(a,b) = ab2, a,b C.
It is continuous and convex with respect to its first variable. Therefore, by Theorem 20, there is aunique minimizer for the regularization network:
minfB
jNn
 f (x j) y j2 + f2B . (62)
We next consider solving the minimizer. To this end, we need the notion of strict convexity ofa normed vector space. We call a normed vector space V strictly convex if whenever x + yV =xV +yV for some x,y 6= 0, there must hold y = x for some > 0. Note that a uniformly convexnormed vector space is automatically strictly convex. The following result was observed in Giles(1967).
Lemma 21 An s.i.p. space V is strictly convex if and only if whenever [x,y]V = xVyV for somex,y 6= 0 there holds y = x for some > 0.
A technical result about s.i.p. spaces is required for solving the minimizer.
Lemma 22 Let V be a strictly convex s.i.p. space. Then for all u,v V
u+ v2V u2V 2Re [v,u]V 0.
Proof Assume that there exist u,v V such that
u+ v2V u2V 2Re [v,u]V < 0.
Then we have u,v 6= 0. We proceed by the above inequality and properties of semiinnerproductsthat
u2V = [u+ v v,u]V = [u+ v,u]V [v,u]V= Re [u+ v,u]V Re [v,u]V [u+ v,u]V  Re [v,u]V
u+ vVuV u+ v2V u
2V
2.
The last inequality above can be simplified as
u+ v2V +u2V 2u+ vVuV .
2764

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Thus, we must have
u+ vV = uV and [u+ v,u]V = u+ vVuV .
Applying the strict convexity of V and Lemma 21, we obtain that u+ v = u, namely, v = 0. This isimpossible.
Theorem 23 (Representer Theorem) Let f0 be the minimizer of (62). Then there exists some c Cn
such thatf 0 =
jNn
c jG(,x j). (63)
If Gx is linearly independent then the right hand side of the above equation is the minimizer if andonly if
ck +
[G(,xk),
jNn
c jG(,x j)
]
B= yk, k Nn. (64)
Proof Let g B . Define the function : R R by (t) := Ez,( f0 + tg), t R. We compute byLemma 7 that
(t) = 2Re jNn
g(x j)( f0(x j) y j + tg(x j))+2Re [g, f0 + tg]B .
Since f0 is the minimizer, t = 0 is the minimum point of . Hence (0) = 0, that is,
jNn
g(x j) f0(x j) y j +[g, f0]B = 0, for all g B.
The above equation can be rewritten as
jNn
[ f0(x j) y jG(,x j),g]B +[ f
0 ,g
]B = 0, for all g B,
which is equivalent to f 0 =
jNn
y j f0(x j)G(,x j). (65)
Therefore, f 0 has the form (63).If Gx is linearly independent then f 0 = jNn c jG(,x j) satisfies (65) if and only if (64) holds.
Thus, it remains to show that condition (65) is enough to ensure that f0 is the minimizer. To thisend, we check that (65) leads to
Ez,( f0 +g)Ez,( f0) = f0 +g2B f0
2B 2Re [g, f0]B +
jNn
g(x j)2,
which by Lemma 22 is nonnegative. The proof is complete.
By Theorem 23, if Gx is linearly independent then the minimizer of (62) can be obtained bysolving (64), which has a unique solution in this case. Using the feature map, the system (64) hasthe following form
ck +
[(xk),
jNn
c j(x j)]
B= yk, k Nn.
As remarked before, this is in general nonlinear about c.
2765

ZHANG, XU AND ZHANG
5.3 Support Vector Machines
In this subsection, we assume that all the spaces are over the field R of real numbers, and considerlearning a classifier from the data z := {(x j,y j) : j Nn} X {1,1}. We shall establish forthis task two learning algorithms in RKBS whose RKHS versions are wellknown (Evgeniou etal., 2000; Scholkopf and Smola, 2002; ShaweTaylor and Cristianini, 2004; Vapnik, 1998; Wahba,1999).
5.3.1 SOFT MARGIN HYPERPLANE CLASSIFICATION
We first focus on the soft margin hyperplane classification by studying
inf
{12w2
W+C1(Nn) : w W , := ( j : j Nn) R
n+, b R
}(66)
subject toy j([(x j),w]W +b) 1 j, j Nn.
Here, C is a fixed positive constant controlling the tradeoff between margin maximization and training error minimization. If a minimizer (w0,0,b0) W Rn+ R exists, the classifier is taken assgn([(),w0]W +b0).
Recall by Theorem 10 that functions in B are of the form
f = [(),w]W , w W (67)
and f B = wW .
Introduce the loss function
Lb(a,y) := max{1ayby,0}, (a,y) R{1,1}, b R,
and the error functional on B,
Eb,z,( f) := f 2B +
jNn
Lb( f(x j),y j), f
B
where := 1/(2C). Then we observe that (66) can be equivalently rewritten as
inf{Eb,z,( f) : f B, b R}. (68)
When b = 0, (68) is also called the support vector machine classification (Wahba, 1999).If a minimizer ( f 0 ,b0) B
R for (68) exists then by (67), the classifier followed from (66)will be taken as sgn( f 0 + b0). It can be verified that for every b R, Lb is convex and continuouswith respect to its first variable. This enables us to prove the existence of minimizers for (68) basedon Theorem 20.
Proposition 24 Suppose that {y j : j Nn} = {1,1}. Then there exists a minimizer ( f 0 ,b0) BR for (68). Moreover, the first component f 0 of the minimizer is unique.
2766

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Proof The uniqueness of f 0 follows from the fact that Eb,z, is strictly convex with respect to itsfirst variable. It remains to deal with the existence. Let e be the infimum (68). Then there exists asequence ( f k ,b
k) B
R, k N such that
limk
Ebk,z,( fk ) = e. (69)
Since {y j : j Nn} = {1,1}, {bk : k N} is bounded on R. We may hence assume that bkconverges to some b0 R. By Theorem 20, min{Eb0,z,( f
) : f B} has a unique minimizer f 0 .The last fact we shall need is the simple observation that
max{1ayby,0}max{1ayby,0} bb for all a,b,b R and y {1,1}.
Thus, we get for all k N that
Eb0,z,( f0 ) Eb0,z,( f
k ) Ebk,z,( f
k )+nb0 bk,
which together with (69) and limk
bk = b0 implies that ( f 0 ,b0) is a minimizer for (68).
Since the soft margin hyperplane classification (66) is equivalent to (68), we obtain by Proposition 24 that it has a minimizer (w0,0,b0) W Rn+R, where the first component w0 is unique.
We shall prove a representer theorem for f 0 using the following celebrated geometric consequence of the HahnBanach theorem (see, Conway, 1990, page 111).
Lemma 25 Let A be a closed convex subset of a normed vector space V and b a point in V that isnot contained in A. Then there exist T V and R such that
T (b) < < T (a), for all a A.
Theorem 26 (Representer Theorem) Let f 0 be the minimizer of (68). Then f0 lies in the closedconvex cone coGz spanned by Gz := {y jG(x j, ) : j Nn}, that is, there exist j 0 such that
f0 = jNn
jy jG(x j, ). (70)
Consequently, the minimizer w0 of (66) belongs to the closed convex cone spanned by y j(x j),j Nn.
Proof Assume that f0 / coGz. By Lemmas 25 and 8, there exists g B and R such that for all 0
[ f0,g]B < < [y jG(x j, ),g]B = y jg(x j), j Nn.
Choosing = 0 above yields that < 0. That y jg(x j) > for all 0 implies y jg(x j) 0,j Nn. We hence obtain that
[ f0,g]B < 0 y jg(x j), j Nn.
We choose f = f 0 + tg, t > 0. First, observe from y jg(x j) 0 that
1 y j f(x j) 1 y j f
0 (x j), j Nn. (71)
2767

ZHANG, XU AND ZHANG
By Lemma 7,
limt0+
f 0 + tg2
B f 0
2B
t= 2[g, f 0 ]B = 2[ f0,g]B < 0.
Therefore, there exists t > 0 such that f 2B
< f 0 2B
. This combined with (71) implies thatEb,z,( f ) < Eb,z,( f 0 ) for every b R, contradicting the hypothesis that f
0 is the minimizer.
To solve (68), by Theorem 26 one substitutes equation (70) into (68) to obtain a convex optimization problem about j subject to the constraint that j 0, j Nn.
5.3.2 HARD MARGIN HYPERPLANE CLASSIFICATION
Consider in the feature space W the following hard margin classification problem
inf{wW : w W , b R} (72)
subject toy j([(x j),w]W +b) 1, j Nn.
Provided that the minimizer (w0,b0) W R exists, the classifier is sgn([(),w0]W +b0).Hard margin classification in s.i.p. spaces was discussed in Der and Lee (2007). Applying
the results in our setting tells that if b is fixed then (72) has a unique minimizer w0 and w0 co{y j(x j) : j Nn}. As a corollary of Proposition 24 and Theorem 26, we obtain here that if{y j : j Nn} = {1,1} then (72) has a minimizer (w0, b0), where w0 is unique and belongs to theset co{y j(x j) : j Nn}.
We draw the conclusion that the support vector machine classifications in this subsection allreduce to a convex optimization problem.
5.4 Kernel Principal Component Analysis
Kernel principal component analysis (PCA) plays a foundational role in data preprocessing forother learning algorithms. We shall present an extension of kernel PCA for RKBS. To this end, letus briefly review the classical kernel PCA (see, for example, Scholkopf and Smola, 2002; Scholkopfet al., 1998).
Let x := {x j : j Nn} X be a set of inputs. We denote by d(w,V ) the distance from w Wto a closed subspace V of W . Fix m N. For each subspace V W with dimension dimV = m,we define the distance from V to (x) as
D(V,(x)) :=1n jNn
(d((x j),V ))2.
Suppose that {u j : j Nm} is a basis for V . Then for each v W the best approximation v0 inV of v exists. Assume that v0 = jNm ju j, j C, j Nm. By Lemma 17, the coefficients jsare uniquely determined by
[uk,v
jNm
ju j
]
W
= 0, k Nm. (73)
2768

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
In the case when W is a Hilbert space, the system (73) of equations resulting from best approximation is linear about js. This enables us to construct a unique mdimensional subspace V0 Wsuch that
D(V0,(x)) = min{D(V,(x)) : V W subspace, dimV = m}. (74)
Let T be the compact operator on W determined by
(Tu,v)W =1n jNn
(u,(x j))W ((x j),v)W , u,v W . (75)
We let v j, j Nm be the unit eigenvectors of T corresponding to its first m largest eigenvalues. Thenv js form an orthonormal basis for V0 and are called the principal components of (x). For eachx X , (((x),v j)W : j Nm) Cm is its new feature. Therefore, kernel PCA amounts to selectingthe new feature map from X to Cm. The dimension m is usually chosen to be much smaller than theoriginal dimension of W . Moreover, by (74), the new features of x are expected to become sparserunder this mapping.
The analysis of PCA in Hilbert spaces breaks in s.i.p. spaces where (73) is nonlinear. To tacklethis problem, we suggest using a class of linear functionals to measure the distance between twoelements in W . Specifically, we choose B W and set for all u,v W
dB(u,v) :=
(
bB
(u v,b)W 2
)1/2.
The idea is that if dB(u,v) is small for a carefully chosen set B of linear functionals then u vWshould be small, and vice versa. In particular, if W is a Hilbert space and B is an orthonormal basisfor W then dB(u,v) = u vW . From the practical consideration, we shall use what we have athand, that is, (x). Thus, we define for each u,v W
d(x)(u,v) :=
(
jNn
[u v,(x j)]W2)1/2
.
This choice of distance is equivalent to mapping X into Cn by
(x) := ([(x),(x j)]W : j Nn), x X .
Consequently, new features of elements in X will be obtained by applying the classical PCA to(x j), j Nn in the Hilbert space Cn.
In our method the operator T defined by (75) on Cn is of the form
Tu =1n jNn
((x j)u)(x j), u Cn,
where (x j) is the conjugate transpose of (x j). One can see that T has the matrix representationTu = Mxu, u Cn, where
Mx :=1n(G[x]G[x])T .
2769

ZHANG, XU AND ZHANG
Let k, k Nn, be the eigenvalues of Mx arranged in nondecreasing order. We find for each k Nmthe unit eigenvector k := (kj : j Nn) Cn corresponding to k, that is,
Mxk = kk.
Vectors k, k Nm, form an orthonormal sequence. The new feature for x X is hence
(((x),k)Cn : k Nm) Cm.
We compute explicitly that
((x),k)Cn = jNn
kjG(x,x j), k Nm.
We remark that unlike the previous three learning algorithms, the kernel PCA presented here onlymakes use of the kernel G and is independent of the semiinnerproduct on W . The kernel trick canhence be applied to this algorithm.
6. Conclusion
We have introduced the notion of reproducing kernel Banach spaces and generalized the correspondence between an RKHS and its reproducing kernel to the setting of RKBS. S.i.p. RKBS werespecially treated by making use of semiinnerproducts and the duality mapping. A semiinnerproduct shares many useful properties of an inner product. These properties and the general theoryof semiinnerproducts make it possible to develop many learning algorithms in RKBS. As illustration, we discussed in the RKBS setting the minimal norm interpolation, regularization network,support vector machines, and kernel PCA. Various representer theorems were established.
This work attempts to provide an appropriate mathematical foundation of kernel methods forlearning in Banach spaces. Many theoretical and practical issues are left for future research. An immediate challenge is to construct a class of useful RKBS and the corresponding reproducing kernels.By the classical theory of RKHS, a function K is a reproducing kernel if and only if the finite matrix(1) is always hermitian and positive semidefinite. This function property characterization bringsgreat convenience to the construction of positive definite kernels. Thus, we ask what characteristicsa function must possess so that it is a reproducing kernel for some RKBS. Properties of RKBS andtheir reproducing kernels also deserve a systematic study. For the applications, we have seen thatminimum norm interpolation and regularization network reduce to systems of nonlinear equations.Dealing with the nonlinearity requires algorithms specially designed for the underlying s.i.p. space.On the other hand, support vector machines can be reformulated into certain convex optimizationproblems. Finally, section 5.4 only provides a possible implementation of kernel PCA for RKBS.We are interested in further careful analysis and efficient algorithms for these problems. We shallreturn to these issues in future work.
Acknowledgments
Yuesheng Xu was supported in part by the US National Science Foundation under grant DMS0712827. Jun Zhang was supported in part by the US National Science Foundation under grant
2770

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
0631541. This research was accomplished when the last author (J.Z.) was on leave from the University of Michigan to AFOSR under an IPA assignment.
Send all correspondence to Yuesheng Xu.
Appendix A.
In this appendix, we provide proofs of two results stated in the previous sections of this paper. Thefirst one is about the minimization problem (5) in the introduction.
Proposition 27 If : Rd [0,+) is strictly concave and > 0, then every minimizer of
min{(x)+x1 : x Rd} (76)
has at most one nonzero element.
Proof Assume to the contrary that x0 Rd is a minimizer of (76) with more than one nonzeroelements. Then x0 is not an extreme points of the sphere {x Rd : x1 = x01}. In other words,there exist two distinct vectors x1,x2 Rd and some (0,1) such that
x0 = x1 +(1)x2 and x11 = x21 = x01 .
By the strict concavity of , we get that
(x0)+x01 > (x1)+(1)(x2)+x01= ((x1)+x11)+(1)((x2)+x21).
Therefore, we must have either
(x0)+x01 > (x1)+x11
or
(x0)+x01 > (x2)+x21 .
Either case contradicts the hypothesis that x0 is a minimizer of (76).
The second result confirms that (27) indeed defines a semiinnerproduct.
Proposition 28 Let V be a normed vector space over C. If for all x,y V \ {0} the limit
limtR, t0
x+ tyV xVt
exists then [, ]V : V V C defined by
[x,y]V := yV
(lim
tR, t0
y+ txV yVt
+ i limtR, t0
iy+ txV yVt
)if x,y 6= 0 (77)
and [x,y]V := 0 if x = 0 or y = 0 is a semiinnerproduct on V .
2771

ZHANG, XU AND ZHANG
Proof First, we obtain for x 6= 0 that
[x,x]V = xV
(lim
tR, t0
(1+ t)xV xVt
+ i limtR, t0
(i+ t)xV xVt
)
= x2V
(lim
tR, t0
1+ t1t
+ i limtR, t0
i+ t1t
)
= x2V (1+0) = x2V > 0.
(78)
We then deal with the remaining three conditions of a semiinnerproduct. Clearly, they are trueif one of the arguments involved is the zero vector. Let x,y,z V \{0}. We start with the estimate:
Re [x+ y,z]V = zV limtR, t0+
z+ tx+ tyV zVt
zV limtR, t0+
z2 + txV +z2 + tyV zVt
= zV
(lim
tR, t0+
z2 + txV z2V
t+ lim
tR, t0+
z2 + tyV z2V
t
)
= zV
(lim
tR, t0+
z+2txV zV2t
+ limtR, t0+
z+2tyV zV2t
).
The above equation implies that
Re [x+ y,z]V Re [x,z]V + Re [y,z]V . (79)
It can be easily verified that [x,y]V = [x,y]V . Replacing y with x+ y, and x with x in the aboveequation yields that
Re [y,z]V Re [x,z]V + Re [x+ y,z]V = Re [x,z]V + Re [x+ y,z]V . (80)
Combining (79) and (80), we get that Re [x+ y,z]V = Re [x,z]V + Re [y,z]V . Similar arguments leadto that Im [x+ y,z]V = Im [x,z]V + Im [y,z]V . Therefore,
[x+ y,z]V = [x,z]V +[y,z]V . (81)
Next we see for all R\{0} that
[x,y]V = yV limtR, t0
y+ txV yVt
= yV limtR, t0
y+ txV yVt
= [x,y]V .
It is also clear from the definition (77) that [ix,y]V = i[x,y]V . We derive from these two facts and(81) for every = + i, , R that
[x,y]V = [x+ ix,y]V = [x,y]V +[ix,y]V = [x,y]V + i[x,y]V= [x,y]V + i[x,y]V = (+ i)[x,y]V = [x,y]V .
(82)
We then proceed for C\{0} by (82) that
[x,y]V = yV limtR, t0
y+ txV yVt
= yV  limtR, t0
y+ t xV yVt
= 2[x,y]V =
2
[x,y]V = [x,y]V .
(83)
2772

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
Finally, we find some C such that  = 1 and [x,y]V = [x,y]V , and then obtain by (82) and(77) that
[x,y]V  = [x,y]V = [x,y]V = yV limtR, t0+
y+ txV yVt
yV limtR, t0+
yV + txV yVt
= yVxV = xVyV .
By (78), the above inequality has the equivalent form
[x,y]V  [x,x]1/2V [y,y]
1/2V . (84)
Combining Equations (78), (81), (82), (83), and (84) proves the proposition.
References
A. Argyriou, C. A. Micchelli and M. Pontil. When is there a representer theorem? Vector versusmatrix regularizers. arXiv:0809.1590v1.
N. Aronszajn. Theory of reproducing kernels. Trans. Amer. Math. Soc., 68: 337404, 1950.
K. P. Bennett and E. J. Bredensteiner. Duality and geometry in SVM classifiers. In Proceeding of theSeventeenth International Conference on Machine Learning, pages 5764, Morgan Kaufmann,San Francisco, 2000.
S. Canu, X. Mary and A. Rakotomamonjy. Functional learning through kernel. In Advances inLearning Theory: Methods, Models and Applications, pages 89110, NATO Science Series III:Computer and Systems Sciences, Volume 190, IOS Press, Amsterdam, 2003.
A. Caponnetto, C. A. Micchelli, M. Pontil and Y. Ying. Universal multitask kernels. Journal ofMachine Learning Research, 9: 16151646, 2008.
J. B. Conway. A Course in Functional Analysis. 2nd Edition, SpringerVerlag, New York, 1990.
F. Cucker and S. Smale. On the mathematical foundations of learning. Bull. Amer. Math. Soc., 39:149, 2002.
D. F. Cudia. On the localization and directionalization of uniform convexity. Bull. Amer. Math. Soc.,69: 265267, 1963.
R. Der and D. Lee. Largemargin classification in Banach spaces. JMLR Workshop and ConferenceProceedings, 2: AISTATS: 9198, 2007.
T. Evgeniou, M. Pontil and T. Poggio. Regularization networks and support vector machines. Adv.Comput. Math., 13: 150, 2000.
M. J. Fabian, P. Habala, P. Hajek and J. Pelant. Functional Analysis and InfiniteDimensional Geometry. Springer, New York, 2001.
2773

ZHANG, XU AND ZHANG
C. Gentile. A new approximate maximal margin classification algorithm. Journal of Machine Learning Research, 2: 213242, 2001.
J. R. Giles. Classes of semiinnerproduct spaces. Trans. Amer. Math. Soc., 129: 436446, 1967.
M. Hein, O. Bousquet and B. Scholkopf. Maximal margin classification for metric spaces. J. Comput. System Sci., 71: 333359, 2005.
V. I. Istratescu. Strict Convexity and Complex Strict Convexity: Theory and Applications. LectureNotes in Pure and Applied Mathematics 89, Marcel Dekker, New York, 1984.
D. Kimber and P. M. Long. Online learning of smooth functions of a single variable. Theoret.Comput. Sci., 148: 141156, 1995.
G. Kimeldorf and G. Wahba. Some results on Tchebycheffian spline functions. J. Math. Anal. Appl.,33: 8295, 1971.
G. Lumer. Semiinnerproduct spaces. Trans. Amer. Math. Soc., 100: 2943, 1961.
J. Mercer. Functions of positive and negative type and their connection with the theorey of integralequations. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci., 209: 415446, 1909.
C. A. Micchelli and M. Pontil. A function representation for learning in Banach spaces. In LearningTheory, pages 255269, Lecture Notes in Computer Science 3120, Springer, Berlin, 2004.
C. A. Micchelli and M. Pontil. On learning vectorvalued functions. Neural Comput., 17: 177204,2005.
C. A. Micchelli and M. Pontil. Feature space perspectives for learning the kernel. Machine Learning,66: 297319, 2007.
C. A. Micchelli, Y. Xu and P. Ye. Cucker Smale learning theory in Besov spaces. In Advances inLearning Theory: Methods, Models and Applications, pages 4768, IOS Press, Amsterdam, TheNetherlands, 2003.
C. A. Micchelli, Y. Xu and H. Zhang. Universal kernels. Journal of Machine Learning Research, 7:26512667, 2006.
C. A. Micchelli, Y. Xu and H. Zhang. Optimal learning of bandlimited functions from localizedsampling. J. Complexity, 25: 85114, 2009.
W. Rudin. Real and Complex Analysis. 3rd Edition, McGrawHill, New York, 1987.
S. Saitoh. Integral Transforms, Reproducing Kernels and Their Applications. Pitman ResearchNotes in Mathematics Series 369, Longman, Harlow, 1997.
B. Scholkopf, R. Herbrich and A. J. Smola. A generalized representer theorem. In Proceeding of the14th Annual Conference on Computational Learning Theory and the 5th European Conferenceon Computational Learning Theory, pages 416426, SpringerVerlag, London, UK, 2001.
B. Scholkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization,Optimization, and Beyond. MIT Press, Cambridge, Mass, 2002.
2774

REPRODUCING KERNEL BANACH SPACES FOR MACHINE LEARNING
B. Scholkopf, A. J. Smola and K.R. Muller. Nonlinear component analysis as a kernel eigenvalueproblem. Neural Comput., 10: 12991319, 1998.
J. ShaweTaylor and N. Cristianini. Kernel Methods for Pattern Analysis. Cambridge UniversityPress, Cambridge, 2004.
I. Steinwart. On the influence of the kernel on the consistency of support vector machines. Journalof Machine Learning Research, 2: 6793, 2001.
J. A. Tropp. Just relax: convex programming methods for identifying sparse signals in noise. IEEETrans. Inform. Theory, 52: 10301051, 2006.
V. N. Vapnik. Statistical Learning Theory. Wiley, New York, 1998.
U. von Luxburg and O. Bousquet. Distancebased classification with Lipschitz functions. Journalof Machine Learning Research, 5: 669695, 2004.
G. Wahba. Support vector machines, reproducing kernel Hilbert spaces and the randomized GACV.In Advances in Kernel MethodsSupport Vector Learning, pages 6986, MIT Press, Cambridge,Mass, 1999.
Y. Xu and H. Zhang. Refinable kernels. Journal of Machine Learning Research, 8: 20832120,2007.
Y. Xu and H. Zhang. Refinement of reproducing kernels. Journal of Machine Learning Research,10: 107140, 2009.
T. Zhang. On the dual formulation of regularized linear systems with convex risks. Machine Learning, 46: 91129, 2002.
D. Zhou, B. Xiao, H. Zhou and R. Dai. Global geometry of SVM classifiers. Technical Report30502, Institute of Automation, Chinese Academy of Sciences, 2002.
2775