Topics

Review of lecture 2/11Error, Residual and Condition Number

Review of lecture 2/16Backward Error AnalysisThe General Case

1 / 22

Theorem (Calculation of 2 norm of a symmetric matrix)If A = At is symmetric then ‖A‖2 is given by

‖A‖2 = max{|λ| : λ is an eigenvalue of A}.

Theorem (Matrix 1-norm and∞-norm)We have

‖A‖∞ = max1≤i≤N

N∑j=1

|aij |,

‖A‖1 = max1≤j≤N

N∑i=1

|aij |.

2 / 22

Topics

Review of lecture 2/11Error, Residual and Condition Number

Review of lecture 2/16Backward Error AnalysisThe General Case

3 / 22

? Approximate solution x̂ of Ax = b

error : = e = x − x̂ ,residual : = r = b − Ax̂ .

? Small residual need not imply small errorthe point P = (0.5,0.7) and the 2× 2 linear system

x − y = 0−0.8x + y = 1/2.

? Plot is two almost-parallel lines with x̂ between them, far from thesolution.

4 / 22

Justification of picture

I Normalize each equation in the system so that√

a211 + a2

12 = 1

I Normal distance from x̂ to the line is r1 = b1 − a11x̂1 − a12x̂2

I Hence ‖r‖2 =square root of sum of squares of normal distancefrom x̂ to lines.

I Same for 3D and higher dimensions

5 / 22

Recall qualitative facts

I If A is invertible then r = 0 if and only if e = 0.I If A is well conditioned then ||r || and ||e|| are comparable in size.I If A is ill conditioned then ||r || can be small while ||e|| is large.I det(A) is not the right way to quantify this connection.

6 / 22

Condition number

DefinitionLet ‖ · ‖ be a matrix norm. Then the condition number of the matrix Ainduced by the vector norm ‖ · ‖ is

cond‖·‖(A) := ‖A−1‖‖A‖.

? Usually use κ

κ(A) = cond(A) = condition number of A.

7 / 22

Matlab

I cond(A) or cond(A,p)I condest(A)I rcond(A) = reciprocal condition number

8 / 22

Theorem (Relative Error ≤ cond(A)×Relative Residual)Let Ax = b and let x̂ be an approximation to the solution x. Withr = b − Ax̂

‖x − x̂‖‖x‖

≤ cond(A)‖r‖‖b‖

.

9 / 22

Proof.I Ae = r so e = A−1r

‖e‖ = ‖A−1r‖ ≤ ‖A−1‖‖r‖. (1)

I Ax = b so ‖b‖ = ‖Ax‖ ≤ ‖A‖‖x‖.I Dividing the smaller side of (1) by the larger quantity and the

larger side of (1) by the smaller gives

‖e‖‖A‖‖x‖

≤ ‖A−1‖‖r‖‖b‖

.

I Rearrangement proves the theorem.

10 / 22

I ‖I‖ = 1 so cond(I) = 1 in any induced matrix norm. Similarly foran orthogonal matrix cond2(O) = 1.

I Example: error = (cond(A))*residual

11 / 22

Topics

Review of lecture 2/11Error, Residual and Condition Number

Review of lecture 2/16Backward Error AnalysisThe General Case

12 / 22

Example (The Hilbert Matrix)The N × N Hilbert matrix HN×N is the matrix with entries

Hij =1

i + j − 1, 1 ≤ i , j ≤ n.

This matrix is extremely ill conditioned even for quite moderate valueson n.

1 12

13

14

12

13

14

15

13

14

15

16

14

15

16

17

13 / 22

Topics

Review of lecture 2/11Error, Residual and Condition Number

Review of lecture 2/16Backward Error AnalysisThe General Case

14 / 22

[Basic Result of Backward Error Analysis] The result x̂ ofsolving Ax = b by Gaussian elimination in finite precisionarithmetic subject to rounding errors is precisely the same asthe exact solution of a perturbed problem

(A + E)x̂ = b + f (2)

where‖E‖||A||

,‖f‖||b||

= O(machine precision).

15 / 22

Theorem (Effect of Storage errors in A)Let A be an N × N matrix. Suppose Ax = b and (A + E)x̂ = b. Then,

‖x − x̂‖‖x̂‖

≤ cond(A)‖E‖‖A‖

.

Theorem (Effect of Storage Errors in b)Let Ax = b and Ax̂ = b + f . Then

‖x − x̂‖‖x‖

≤ cond(A)‖f‖‖b‖

.

16 / 22

Sketch proof Theorem 145Proof.Step 1: By subtraction get an equation for the error driven by theperturbation:

Ax = b(A + E)x̂ = b

___subtract____A(x − x̂) = Ex̂

x − x̂ = A−1Ex̂

Step 2: Take norms

||x − x̂ || = ||A−1Ex̂ || ≤ ||A−1|| · ||E || · ||x̂ || ‖A‖‖A‖

Step 3: rearrange using def of cond no

‖x − x̂‖‖x̂‖

≤ cond(A)‖E‖‖A‖

.

17 / 22

Sketch proof Theorem 146

Proof.I Since Ax̂ = Ax + f , x − x̂ = −A−1fI ‖x − x̂‖ ≤ ‖A−1‖‖f‖ = cond(A) ‖f‖‖A‖I ‖x − x̂‖ ≤ cond(A) ‖f‖‖b‖‖x‖ because ‖b‖ ≤ ‖A‖‖x‖.

18 / 22

Facts about condition number

I When E is due to roundoff errors ‖E‖/‖A‖ = O(machineprecision). cond(A) tells you how many significant digits arelost (worst case) when solving Ax = b

I cond(A) ≥ 1 and cond(I) = 1.I Scaling A does not influence cond(A):I cond(A) depends on the norm chosen but usually it is of the

same order of magnitude for different norms.I If A is symmetric then

cond2(A) = |λ|max/|λ|min.

I If A is symmetric, positive definite and ‖ · ‖ = ‖ · ‖2, then cond(A)equals the spectral condition number, λmax/λmin

cond2(A) = λmax/λmin.

I cond(A) = cond(A−1).

19 / 22

Topics

Review of lecture 2/11Error, Residual and Condition Number

Review of lecture 2/16Backward Error AnalysisThe General Case

20 / 22

Working toward the general case

Lemma (Spectral localization)For any N × N matrix B and ‖ · ‖ any matrix norm:

|λ(B)| ≤ ‖B‖.

Proof.For an eigenpair (λ, φ), Bφ = λφ.Thus |λ|‖φ‖ = ‖Bφ‖ ≤ ‖B‖‖φ‖.

21 / 22

Theorem 149

Theorem

limn→∞

Bn = 0 if and only if there exists a norm ‖ · ‖ with ‖B‖ < 1.

Proof.I We prove ‖B‖ < 1⇒ ‖Bn‖ ≤ ‖B‖n → 0I Start the induction with n = 2,

‖B2‖ = ‖B · B‖ ≤ ‖B‖ · ‖B‖ ≤ ‖B‖2.

I Assume ‖Bn‖ ≤ ‖B‖n, then

‖Bn+1‖ = ‖BnB‖ ≤ ‖Bn‖‖B‖ ≤ ‖B‖n+1

I ‖Bn‖ ≤ ‖B‖n → 0 as n→∞.

22 / 22