Best approximation in the 2-norm

Elena Celledoni

Department of Mathematical Sciences, NTNU

september 26th 2012

Elena Celledoni NA

Vector space

A real vector space V is a set with a 0 element and threeoperations:• Addition: x , y ∈ V then x + y ∈ V

• Inverse: for all x ∈ V there is −x ∈ V such that x + (−x) = 0• Scalar multiplication: λ ∈ R, x ∈ V then λ x ∈ V

1 x + y = y + x

2 (x + y) + z = x + (y + z)

3 0 + x = x + 0 = x4 0 x = 05 1 x = x

6 (λµ) x = λ (µ x)

7 λ (x + y) = λx + λy

8 (λ+ µ) x = λx + µx

Elena Celledoni NA

Inner product spaces: 9.2

Definition of inner product spaceLet V be a linear space over R.A function 〈·, ·〉 : V × V → R, is called an inner product on V if:

1 〈f + g , h〉 = 〈f , h〉+ 〈g , h〉 for all f , g , h in V ;2 〈λf , g〉 = λ〈f , g〉 for all f , g in V and λ ∈ R;3 〈f , g〉 = 〈g , f 〉 for all f , g in V ;4 〈f , f 〉 > 0 if f 6= 0, f ∈ V .

A linear space with an inner product is an inner product space.

Example

The n-dimensional Euclidean space Rn is an inner product space

〈u,b〉 =n∑

i=0

uivi , u, v ∈ Rn

and u = [u1, . . . , un]T , v = [v1, . . . , vn]T .

Elena Celledoni NA

Inner product spaces: 9.2

Definition of inner product spaceLet V be a linear space over R.A function 〈·, ·〉 : V × V → R, is called an inner product on V if:

1 〈f + g , h〉 = 〈f , h〉+ 〈g , h〉 for all f , g , h in V ;2 〈λf , g〉 = λ〈f , g〉 for all f , g in V and λ ∈ R;3 〈f , g〉 = 〈g , f 〉 for all f , g in V ;4 〈f , f 〉 > 0 if f 6= 0, f ∈ V .

A linear space with an inner product is an inner product space.

Example

The n-dimensional Euclidean space Rn is an inner product space

〈u,b〉 =n∑

i=0

uivi , u, v ∈ Rn

and u = [u1, . . . , un]T , v = [v1, . . . , vn]T .

Elena Celledoni NA

Inner product spaces: 9.2

Definition of inner product spaceLet V be a linear space over R.A function 〈·, ·〉 : V × V → R, is called an inner product on V if:

1 〈f + g , h〉 = 〈f , h〉+ 〈g , h〉 for all f , g , h in V ;2 〈λf , g〉 = λ〈f , g〉 for all f , g in V and λ ∈ R;3 〈f , g〉 = 〈g , f 〉 for all f , g in V ;4 〈f , f 〉 > 0 if f 6= 0, f ∈ V .

A linear space with an inner product is an inner product space.

Example

The n-dimensional Euclidean space Rn is an inner product space

〈u,b〉 =n∑

i=0

uivi , u, v ∈ Rn

and u = [u1, . . . , un]T , v = [v1, . . . , vn]T .

Elena Celledoni NA

Inner product spaces: orthogonality and norm

OrthogonalityIf in an inner product space V we have

〈f , g〉 = 0

then we say that f is orthogonal to g .

Induced normIn an inner product space we can define the norm

‖f ‖ := 〈f , f 〉12 ,

this is called norm induced by the inner product.

Theorem An inner product space V over R with the inducednorm defined above is a normed space.Proof. Need to prove that the induced norm satisfies the axioms ofnorm, the proof makes use of the Cauchy-Schwarz inequality.

Elena Celledoni NA

Inner product spaces: orthogonality and norm

OrthogonalityIf in an inner product space V we have

〈f , g〉 = 0

then we say that f is orthogonal to g .Induced normIn an inner product space we can define the norm

‖f ‖ := 〈f , f 〉12 ,

this is called norm induced by the inner product.

Theorem An inner product space V over R with the inducednorm defined above is a normed space.Proof. Need to prove that the induced norm satisfies the axioms ofnorm, the proof makes use of the Cauchy-Schwarz inequality.

Elena Celledoni NA

Inner product spaces: orthogonality and norm

OrthogonalityIf in an inner product space V we have

〈f , g〉 = 0

then we say that f is orthogonal to g .Induced normIn an inner product space we can define the norm

‖f ‖ := 〈f , f 〉12 ,

this is called norm induced by the inner product.

Theorem An inner product space V over R with the inducednorm defined above is a normed space.Proof. Need to prove that the induced norm satisfies the axioms ofnorm, the proof makes use of the Cauchy-Schwarz inequality.

Elena Celledoni NA

Lemma: Cauchy-Schwarz inequality

|〈f , g〉| ≤ ‖f ‖ ‖g‖, ∀f , g ∈ V .

Proof Similar to the one seen in chapter 2 for the inner product in Rn.

• Using the lemma we prove the triangular inequality for the inducednorm:

‖f + g‖2 = 〈f + g , f + g〉 = ‖f ‖2 + 2〈f , g〉+ ‖g‖2

from Cauchy-Schwarz we get

‖f + g‖2 ≤ ‖f ‖2 + 2‖f ‖‖g‖+ ‖g‖2 = (‖f ‖+ ‖g‖)2

so‖f + g‖ ≤ ‖f ‖+ ‖g‖.

• The other two axioms of norm are directly proved using thedefinition of induced norm.

Elena Celledoni NA

Lemma: Cauchy-Schwarz inequality

|〈f , g〉| ≤ ‖f ‖ ‖g‖, ∀f , g ∈ V .

Proof Similar to the one seen in chapter 2 for the inner product in Rn.

• Using the lemma we prove the triangular inequality for the inducednorm:

‖f + g‖2 = 〈f + g , f + g〉 = ‖f ‖2 + 2〈f , g〉+ ‖g‖2

from Cauchy-Schwarz we get

‖f + g‖2 ≤ ‖f ‖2 + 2‖f ‖‖g‖+ ‖g‖2 = (‖f ‖+ ‖g‖)2

so‖f + g‖ ≤ ‖f ‖+ ‖g‖.

• The other two axioms of norm are directly proved using thedefinition of induced norm.

Elena Celledoni NA

Lemma: Cauchy-Schwarz inequality

|〈f , g〉| ≤ ‖f ‖ ‖g‖, ∀f , g ∈ V .

Proof Similar to the one seen in chapter 2 for the inner product in Rn.

• Using the lemma we prove the triangular inequality for the inducednorm:

‖f + g‖2 = 〈f + g , f + g〉 = ‖f ‖2 + 2〈f , g〉+ ‖g‖2

from Cauchy-Schwarz we get

‖f + g‖2 ≤ ‖f ‖2 + 2‖f ‖‖g‖+ ‖g‖2 = (‖f ‖+ ‖g‖)2

so‖f + g‖ ≤ ‖f ‖+ ‖g‖.

• The other two axioms of norm are directly proved using thedefinition of induced norm.

Elena Celledoni NA

Space of continuous functions

Example (9.3)

C [a, b] (continuous functions defined on the closed interval [a, b])is an inner product space with the inner product

〈f , g〉 =

∫ b

aw(x)f (x)g(x)dx ,

and w is a weight function defined, positive, continuous andintegrable on (a, b). The induced norm is

‖f ‖2 =

(∫ b

aw(x) |f (x)|2dx

) 12

.

This norm is called “the 2-norm on C [a, b]”.

Elena Celledoni NA

Space of square integrable functions: L2

We do not need f to be continuous in order for ‖f ‖2 to be finite, it issufficient that |f |2w is integrable on [a, b].

ExampleL2w (a, b) := {f : (a, b)→ R |w(x)|f (x)|2 is integrable on (a, b)}.

This is an inner product space with the inner product

〈f , g〉 =

∫ b

a

w(x)f (x)g(x)dx ,

and w is a weight function defined, positive, continuous and integrableon (a, b). The induced norm is

‖f ‖2 =

(∫ b

a

w(x) |f (x)|2dx

) 12

.

This norm is called the L2-norm.Note C [a, b] ⊂ L2

w (a, b). In the case w ≡ 1 then we write L2(a, b).

Elena Celledoni NA

Space of square integrable functions: L2

We do not need f to be continuous in order for ‖f ‖2 to be finite, it issufficient that |f |2w is integrable on [a, b].

ExampleL2w (a, b) := {f : (a, b)→ R |w(x)|f (x)|2 is integrable on (a, b)}.

This is an inner product space with the inner product

〈f , g〉 =

∫ b

a

w(x)f (x)g(x)dx ,

and w is a weight function defined, positive, continuous and integrableon (a, b). The induced norm is

‖f ‖2 =

(∫ b

a

w(x) |f (x)|2dx

) 12

.

This norm is called the L2-norm.Note C [a, b] ⊂ L2

w (a, b). In the case w ≡ 1 then we write L2(a, b).

Elena Celledoni NA

Best approximation in the 2-norm: ch 9.3

The problem of best approximation in the 2-norm can beformulated as follows:

Best approximation problem in the 2-norm

Given that f ∈ L2w (a, b), find pn ∈ Πn such that

‖f − pn‖2 = infq∈Πn

‖f − q‖2

such pn is called a polynomial of best approximation of degreen to the function f in the 2-norm on (a, b).

Theorem 9.2Let f ∈ L2

w (a, b), there exists a unique polynomial pn ∈ Πn suchthat

‖f − pn‖2 = minq∈Πn

‖f − q‖2.

Elena Celledoni NA

Best approximation in the 2-norm: ch 9.3

The problem of best approximation in the 2-norm can beformulated as follows:

Best approximation problem in the 2-norm

Given that f ∈ L2w (a, b), find pn ∈ Πn such that

‖f − pn‖2 = infq∈Πn

‖f − q‖2

such pn is called a polynomial of best approximation of degreen to the function f in the 2-norm on (a, b).

Theorem 9.2Let f ∈ L2

w (a, b), there exists a unique polynomial pn ∈ Πn suchthat

‖f − pn‖2 = minq∈Πn

‖f − q‖2.

Elena Celledoni NA

Example: best approximation in ‖ · ‖2 vs ‖ · ‖∞

Let ε > 0 fixed

f (x) = 1− e−xε , x ∈ [0, 1]

findp2−norm0 and p∞−norm

0

Elena Celledoni NA

In general, to find pn(x) = c0 + c1x + · · ·+ cnxn best

approximation in the two norm on [0, 1]:

SolveM c = b

where M is the Hilbert matrix:

Mj ,k =

∫ 1

0xk+j dx =

1k + j + 1

j , k = 0, . . . , n,

and

bT =

b0...bn

, bj =

∫ 1

0f (x)x j dx .

Elena Celledoni NA

In general, to find pn(x) = c0 + c1x + · · ·+ cnxn best

approximation in the two norm on [0, 1]:

SolveM c = b

where M is the Hilbert matrix:

Mj ,k =

∫ 1

0xk+j dx =

1k + j + 1

j , k = 0, . . . , n,

and

bT =

b0...bn

, bj =

∫ 1

0f (x)x j dx .

Elena Celledoni NA

Alternatively write pn(x) = γ0ϕ0 + γ1ϕ1(x) + · · ·+ γnϕn(x)best approximation in the two norm on [a, b]:

andΠn = span{1, x , . . . , xn} = span{ϕ0, ϕ1 . . . , ϕn},

and find γ0, . . . , γn minimizing

E (γ0, . . . , γn) :=

∫ b

a|f (x)−

n∑j=0

γjϕj |2.

This leads toM γ = b

where γ = [γ0, . . . , γn]T , M is the matrix:

Mj ,k = 〈ϕk , ϕj〉 j , k = 0, . . . , n,

andb = [b0, . . . , bn]T bj = 〈f , ϕj〉.

We’d like M diagonal (i.e. 〈ϕk , ϕj〉 = δk,j ).

Elena Celledoni NA

Alternatively write pn(x) = γ0ϕ0 + γ1ϕ1(x) + · · ·+ γnϕn(x)best approximation in the two norm on [a, b]:

andΠn = span{1, x , . . . , xn} = span{ϕ0, ϕ1 . . . , ϕn},

and find γ0, . . . , γn minimizing

E (γ0, . . . , γn) :=

∫ b

a|f (x)−

n∑j=0

γjϕj |2.

This leads toM γ = b

where γ = [γ0, . . . , γn]T , M is the matrix:

Mj ,k = 〈ϕk , ϕj〉 j , k = 0, . . . , n,

andb = [b0, . . . , bn]T bj = 〈f , ϕj〉.

We’d like M diagonal (i.e. 〈ϕk , ϕj〉 = δk,j ).Elena Celledoni NA

Def 9.4: Orthogonal polynomials

Given a weight function w on (a, b) we say that the sequence ofpolynomials

ϕj , j = 0, 1, . . .

is a system of orthogonal polynomials on (a, b) with respect tow , if each ϕj is of exact degree j and

〈ϕj , ϕk〉 = δj ,k .

To construct a system of orthogonal polynomials we can use aGram-Schmidt process.

Elena Celledoni NA

Gram-Schmidt orthogonalization

Let ϕ0 ≡ 1 and assume ϕj j = 0, . . . , n n ≥ 0 are alreadyorthogonal so

〈ϕk , ϕj〉 =

∫ b

aw(x)ϕk(x)ϕj(x) dx = 0, k , j ∈ {0, . . . , n}, k 6= j

consider

q(x) = xn+1 − a0ϕ0(x)− · · · − anϕn(x) ∈ Πn+1,

and take

aj =〈xn+1, ϕj〉〈ϕj , ϕj〉

, j = 0, . . . , n

THEN

〈q, ϕj〉 = 〈xn+1, ϕj〉 − aj〈ϕj , ϕj〉 = 0,

and we can takeϕn+1 = α q

with α a scaling constant.

Elena Celledoni NA

Gram-Schmidt orthogonalization

Let ϕ0 ≡ 1 and assume ϕj j = 0, . . . , n n ≥ 0 are alreadyorthogonal so

〈ϕk , ϕj〉 =

∫ b

aw(x)ϕk(x)ϕj(x) dx = 0, k , j ∈ {0, . . . , n}, k 6= j

consider

q(x) = xn+1 − a0ϕ0(x)− · · · − anϕn(x) ∈ Πn+1,

and take

aj =〈xn+1, ϕj〉〈ϕj , ϕj〉

, j = 0, . . . , n

THEN

〈q, ϕj〉 = 〈xn+1, ϕj〉 − aj〈ϕj , ϕj〉 = 0,

and we can takeϕn+1 = α q

with α a scaling constant.

Elena Celledoni NA

Gram-Schmidt orthogonalization

Let ϕ0 ≡ 1 and assume ϕj j = 0, . . . , n n ≥ 0 are alreadyorthogonal so

〈ϕk , ϕj〉 =

∫ b

aw(x)ϕk(x)ϕj(x) dx = 0, k , j ∈ {0, . . . , n}, k 6= j

consider

q(x) = xn+1 − a0ϕ0(x)− · · · − anϕn(x) ∈ Πn+1,

and take

aj =〈xn+1, ϕj〉〈ϕj , ϕj〉

, j = 0, . . . , n

THEN

〈q, ϕj〉 = 〈xn+1, ϕj〉 − aj〈ϕj , ϕj〉 = 0,

and we can takeϕn+1 = α q

with α a scaling constant.Elena Celledoni NA

Example (9.6 Legendre polynomials)

(a, b) = (−1, 1) w(x) ≡ 1:

ϕ0(x) ≡ 1ϕ1(x) = x

ϕ2(x) =32x2 − 1

2

ϕ2(x) =52x3 − 3

2x

Example (9.6 Chebishev polynomials)

(a, b) = (−1, 1) w(x) ≡ (1− x2)−1/2

Tn(x) = cos(n arcos(x)), n = 0, 1, . . . .

Elena Celledoni NA

Characterization of the best approximation in the 2-norm

Theorem 9.3

pn ∈ Πn is such that

‖f − pn‖2 = minq∈Πn

‖f − q‖2, f ∈ L2w (a, b),

if and only if〈f − pn, q〉 = 0, ∀q ∈ Πn

Remark: This theorem relates best approximation to orthogonality.

Elena Celledoni NA

ExampleOn (0, 1) and w(x) ≡ 1 on (0, 1) by Gram-Schmidt orthogonalizationwe get

ϕ0 = 1, ϕ1 = x − 12, ϕ2(x) = x2 − x +

16.

So forf (x) = 1− e−

xε , x ∈ [0, 1]

the best approximation p3 is p3(x) = γ0 + γ1 ϕ1(x) + γ2 ϕ2(x)

γ0 =

∫ 1

0(1− e−

xε ) dx ,

γ1 =

∫ 10 (1− e−

xε )(x − 1

2 ) dx∫ 10 (x − 1

2 )2 dx,

γ1 =

∫ 10 (1− e−

xε )(x2 − x + 1

6 ) dx∫ 10 (x2 − x + 1

6 )2 dx,

Elena Celledoni NA

ExampleOn (0, 1) and w(x) ≡ 1 on (0, 1) by Gram-Schmidt orthogonalizationwe get

ϕ0 = 1, ϕ1 = x − 12, ϕ2(x) = x2 − x +

16.

So forf (x) = 1− e−

xε , x ∈ [0, 1]

the best approximation p3 is p3(x) = γ0 + γ1 ϕ1(x) + γ2 ϕ2(x)

γ0 =

∫ 1

0(1− e−

xε ) dx ,

γ1 =

∫ 10 (1− e−

xε )(x − 1

2 ) dx∫ 10 (x − 1

2 )2 dx,

γ1 =

∫ 10 (1− e−

xε )(x2 − x + 1

6 ) dx∫ 10 (x2 − x + 1

6 )2 dx,

Elena Celledoni NA