Ch.10 Numerical Applications

10-1 Least-squares

Start with Self-test 10-1/459.

Linear equation Y = aX + b

Error function: E = D2 = (Y - (aX+b))2

Regression Formula:

Slope → a = (N ΣXY - (ΣX)(ΣY)) / (N ΣX2 - (ΣX)2)

Intercept → b = (ΣY - a (ΣX)) / N

Extra-credit: How meny operations total are required to calculate a and b

according to the fmlas. above? (Nothing is computed twice)

A: a → 6N+3, b → 3

Problems with linear models

Anscombe’s quartet:

These 4 data sets have the same mean, variance for x, variance for y,

median and correlation coeff.! [Wikipedia]

Beyond linear:

Polynomial models

o Solve linear system!

Logarithmic models → Y = b Xa

o Take ln, then use linear regression!

MS Excel functions for linear and logarithmic regression

Spreadsheet available on webpage

10-3 Matrix Operations

Scalar multiplication

Matrix add., sub.

Dot product (vector mult.)

Matrix mult.

Self-test 10-2 / 472

------------------------------------------------------------------------------

10-4 Linear Systems and Gaussian elimination

Linear equation → The only operations allowed are multiplication with

constants and addition. ( Subtraction also allowed: x – y = x + (-1)y.)

A system of eqns. is called linear if all its equations are linear.

Examples of linear and non-linear systems:

Matrix form: Ax = b

(uppercase for matrices, lowercase for vectors)

In this class we’re considering only the case where the # of unknowns

equals the # of eqns. What does this imply about A, x and b?

Even when the # of unknowns equals the # of eqns., there is no guaranteed

unique solution → 3 cases:

Unique solution

No solution (overdetermined)

Infinite # of solutions (underdetermined)

Illustrations for 2x2 systems …

Illustrations for 3x3 systems → see text

Conclusion: A unique solution exists No row of A is linear

combination of other rows No column of A is linear combination of

other columns (square A!)

Important idea: bringing a matrix to (upper) triangular form!

Handout: Bring this matrix to upper triangular form:

----------------------------------------------------------------------------------

Continuing 10-4 Linear Systems and Gaussian elimination

This program shows how to eliminate coefficients from a matrix.

Only the first TWO elementary row operations are performed from the

Gaussian elimination algorithm.

Since no pivoting is performed, we can see very early on the problem that

develops: The original matrix elements were in the range 1..11, but note

the increase in the magnitudes!

Now we let the Gaussian elimination run through the entire matrix:

-100.36

In order for the solution to work, the vector of right-hand-sides b must undergo the

same row operations as the matrix A.

Handout: Write the augmented matrix A’ =

Bring the augmented matrix A’ to upper triangular form:

Calculate z:

Calculate y through backsubstitution in the second eq:

Calculate x through backsubstitution in the first eq:

Changes in the code to implement augmented matrix:

Complexity for triangularizing augmented Nx(N+1) matrix? →O(N3)

Complexity for backsubstitution? →O(N2)

Conclusion: complexity for Gaussian algorithm overall? → O(N3)

Continuing 10-4 Linear Systems and Gaussian elimination

Repeated row operations can lead to numerical instability (see screenshots

above!)

Which part of the Gaussian algorithm is responsible?

Idea: Try to use the largest pivot available (in absolute value), in order to

make the multiplier as small as possible (in absolute value), a.k.a.

pivoting.

o Partial pivoting

o Complete pivoting

New code for partial pivoting:

And finally the payoff:

Since each column has one

and only one pivot, I’m

renaming the column index

PIVOT

Handout: Write the augmented matrix A’ =

Bring the augmented matrix A’ to upper triangular form using partial pivoting:

Compare the triangular matrix to that from last time …

Calculate z:

Calculate y through backsubstitution in the second eq:

Calculate x through backsubstitution in the first eq:

(Is the final solution the same as that from last time?)

Individual work for next time: Solve in notebook the Hand Example on

pp.479-480.

How about total pivoting? … Swapping columns can lead to an even larger pivot, but the price

to pay is that the unknowns get swapped, too. We have to keep track of these swaps, and un-

swap the solution vector at the end.

Empirical conclusion: partial pivoting is good enough in most practical problems. If the matrix

A is particularly ill-conditioned, we can work a little harder and do complete pivoting.

Gauss or Gauss-Jordan? (Not in text)

[Source: http://math.comsci.us/matrix/jordan.shtml ]

Problems with Gauss-Jordan:

Additional numerical instabilities are introduced by the additional

divisions by pivots!

“Clearing” the upper part of the matrix practically triples the # of

operations: approx. N3/3 for Gauss, N3 for Gauss-Jordan .

Suggested additional reading for Gauss and Gauss-Jordan:

Sections 2.1 and 2.2 of “Numerical Recipes” (link on webpage).

This are not “Big-Oh”, since we fill in the exact multipliers, 1/3 and 1, respectively!

Weakness shared by Gauss and Gauss-Jordan algorithms: How to deal

with multiple right-had-side vectors b?

And now the clincher: What if not all the RHSs are not available at the

same time?

More advanced linear solving with the LU decomposition [not in text]

Explaining the concept of decomposition: The matrix is stored as an

expression involving smaller/simpler parts of itself, e.g.

A = LU (LU decomp.)

A = N-1JN (Jordan decomposition)

A = U V (singular-value decomp.)

A = abT (decomposition for rank-1 matrices)

A = AU -1VTA (biconjugate decomp.)

Etc. etc.

The important thing to understand about any decomposition is that, once

the decomposed parts have been calculated, they can be stored and used at

any time to represent the original matrix A. The decomposition need not

be performed multiple times!

Advantage: the RHSs do not need to be all present up-front, as for

Gaussian elim.

Disadvantage:

How do we solve the system Ax = b if A = LU?

o LUx = b

o Call Ux y → Ly = b → solve for y with forward substitution, since

L is Lower triangular

o Once we know y, solve Ux = y as in Gaussian elimination, with

back-substitution.

Conclusion: forward and back-substitution … but they are both O(N2)!

Complexity:

Next time we’ll show that the matrices L and U can be found in O(N3),

actually N3/3, just like Gaussian elimination!

Then it’s just O(N2) for solving Ax = b for each RHS.

[Handout] Consider the following LU decomposition:

L = 1 0 0 U = 2 3 4

3 1 0 0 1 1

2 4 1 0 0 5

Show with pencil and paper how to solve the system Ax = b by forward

substitution, followed by backsubstitution, where

A = LU b = 1

0

2

A note on L’s diagonal:

L and U together have N2 + N non-zero elements. Because the original A

has only N2, we can choose the remaining N freely. Most implementations

of LU decomp. choose to make all diag. elements of L ones. In this case,

we don’t even need two matrices, we can store both L and U in place in

the old matrix A!

Recommended reading:

Section 2.3 of Numerical Recipes (link on webpage)

Wikipedia article (link on webpage)

------------------------------------------------------------------------------