Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A...

60
Uppsala, April 2014 Subspace methods Gerard Sleijpen Department of Mathematics http://www.staff.science.uu.nl/sleij101/

Transcript of Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A...

Page 1: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Uppsala, April 2014

Subspace methods

Gerard SleijpenDepartment of Mathematics

http://www.staff.science.uu.nl/∼sleij101/

Page 2: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Ax = b Ax = λx

Subspace methods

Iterate until sufficiently accurate:

• Expansion. Expand the search subspace Vk.Restart if dim(Vk) is too large.

• Extraction. Extract an appropriate approximatesolution (ϑ,u) from the search subspace.

Practical aspects.

Construct basis v1, . . . ,vk of the search subspace Vk.

Notation. Vk = [v1, . . . , vk]; Vk = span(Vk).

• Compute expansion vector t, “orthogonalise” to vk+1.orthogonalize, A-orthogonalize, bi-orthogonalize, . . .

• Approximate solution uk ≡ Vk yk; compute yk.

Page 3: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 4: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 5: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Krylov subspace expansion

The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

Krylov basis: then t = Avk is an expansion vector.

Examples. For Ax = λx.

• Arnoldi ((harmonic) Ritz extraction).

Lanczos in case A is Hermitian.

Page 6: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Krylov subspace expansion

The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

Krylov basis: then t = Avk is an expansion vector.

Examples. For Ax = λx.

• Arnoldi ((harmonic) Ritz extraction).

Lanczos in case A is Hermitian.

• Shift and Invert Arnoldi:

t = (A− τ I)−1vk to generate a basis

for the search subspace Kk((A− τ I)−1,v1).

Shift and Invert Lanczos for the Hermitian case

Page 7: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Krylov subspace expansion

The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

Krylov basis: then t = Avk is an expansion vector.

Examples. For Ax = λx.

• Arnoldi ((harmonic) Ritz extraction).

Lanczos in case A is Hermitian.

• Shift and Invert Arnoldi:

t = (A− τ I)−1vk to generate a basis

for the search subspace Kk((A− τ I)−1,v1).

SI expansion amplifies components of eigenvectors with

eigenvalue close to the target τ .

Page 8: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 9: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Convergence without subspace acceleration

Shift & Invert uk+1 = (A− τ I)−1uk. (s&i)

Linear convergence towards eigenvector with eigenvalue

closest to τ .

Rayleigh Quotient Iteration

uk+1 = (A− ρk I)−1uk, where ρk ≡ ρ(uk).

Asymptotic quadratic convergence towards an eigenvector.

Asymptotic cubic convergence with two-sided Rayleigh quo-

tient iteration, in particular, if A Hermitian.

Page 10: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 11: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

Page 12: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

+ Fast convergence (RQI)

+ Subspace allows better control of the process.

Search subspace is not a Krylov subspace.

Page 13: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

+ Fast convergence (RQI)

+ Subspace allows better control of the process.

Search subspace is not a Krylov subspace.

[Ruhe 82]Rational Krylov Sequence method allows

different shifts in Shift and Invert (generating a subspace of

rational functions in A times u0, rather than polynomials),

yet it works with Hessenberg matrices (making computa-

tions a little more efficient).

Page 14: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

+ Fast convergence (RQI)

+ Subspace allows better control of the process.

Search subspace is not a Krylov subspace.

[Ruhe 82]Rational Krylov Sequence method allows

efficient computations.

Computational costs

S& I: Solving (A− τ I)uk = uk+1 requires one LU-dec. in total

RQI: Solving (A− ρkI)uk = uk+1 requires one LU-dec. each step

May not be feasible if n large.

Page 15: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 16: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Ax = λx.

If (ϑ,u) is an approximate eigenpair, r ≡ Au− ϑu

(e.g., ϑ = ρ(u)), then the solution t ⊥ u of

(I− uu∗)(A− ϑI)(I− uu∗)t = −r

is the (first order wrt ‖r‖2) correction of u.

[Sleijpen vd Vorst 95] Jacobi-Davidson

Page 17: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Page 18: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

The Jacobi-Davidson correction equation (jd) can be

viewed as a Newton correction equation for the non-linear

problem

F(λ, x) ≡ (1− ‖x‖22,Ax− λx) = (0,0) :

(jd) can be related to the Jacobian of F : Cn+1 → Cn+1.

Page 19: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Note. If ϑ ≈ λ,

then the system (A− ϑI)t = u is ill conditioned.

Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system

(jd) as a system in the space u⊥ is well-conditioned.

Krylov subspace solvers with initial guess x0 = 0 are suit-

able for solving in u⊥.

Better conditioning

• faster convergence of the linear solver(deflation of small eigenvalue),

• more stability.

Page 20: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Note. If ϑ ≈ λ,

then the system (A− ϑI)t = u is ill conditioned.

Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system

(jd) as a system in the space u⊥ is well-conditioned.

There are block-versions to deal with the non-simple eigen-

value case (but the simple eigenvalue version often works

well in the non-simple case).

Block versions require to know the multiplicity in advance.

Block versions are attractive for computational reasons

(exploit parallelism).

Page 21: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Solve approximately with preconditioned iterative linear sol-

ver as GMRES (or MINRES of A∗ = A).

Often a fixed modest number of steps already lead to fast

convergence.

Issue. How many steps are optimal?

Many steps in the inner loop (to solve (jd))

high quality search subspace of low dimension.

Optimal strategy is available in case A = A∗.

Page 22: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ Mt = −r+ βu with β st t ⊥ u

⇔ t = −M−1r+ βM−1u ⊥ u ⇒ β = u∗M−1ru∗M−1u

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Page 23: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Page 24: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Expansion by t = −M−1r (d) Davidson ’75

Expansion by t = −(I− wu∗u∗w)M−1r (o) Olsen ’93

Page 25: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Interpretation. If M = A− ϑI, then u = M−1r. Hence,

effective expansion with w = M−1u: RQI expansion.

Page 26: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Interpretation. If M = A− ϑI, then u = M−1r. Hence,

effective expansion with w = M−1u: RQI expansion.

+ expansion equation (jd) better conditioned then (rqi).

Page 27: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

0 5 10 15 20 25 30 35 4010

−10

10−8

10−6

10−4

10−2

100

102

norm

of JD

resid

ual

number of outer iterations

5 steps GMRES

− Jacobi−Davidson

−. Gen. Davidson

... Inexact Shift & Invert

A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

Page 28: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

0 5 10 15 20 25 30 35 4010

−12

10−10

10−8

10−6

10−4

10−2

100

102

norm

of JD

resid

ual

number of outer iterations

10 steps GMRES

− Jacobi−Davidson

−. Gen. Davidson

... Inexact Shift & Invert

A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

Page 29: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

0 5 10 15 20 25 30 35 4010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100

102

norm

of JD

resid

ual

number of outer iterations

25 steps GMRES

− Jacobi−Davidson

−. Gen. Davidson

... Inexact Shift & Invert

A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

Page 30: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Precondition (jd) by (I− wu∗u∗w)M−1, where w ≡M−1u

With x0 = 0, preconditioned Krylov requires mult. by

(I− wu∗u∗w)M−1(A− ϑI)

with right-hand side vector (I− wu∗u∗w)M−1r.

Page 31: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Precondition (jd) by (I− wu∗u∗w)M−1, where w ≡M−1u

With x0 = 0, preconditioned Krylov requires mult. by

(I− wu∗u∗w)M−1(A− ϑI)

with right-hand side vector (I− wu∗u∗w)M−1r.

Additional costs. (additional to RQI)

per step mult. by I− wu∗u∗w : 1 AXPY, 1 DOT per step

per Krylov run: 1 solve of Mw = u.

Page 32: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Jacobi–Davidson

• Subspace method

+ Accelerated convergence

+ Steering possibilities

+ variety of selection methods

More costly steps

• Expansion vectors from JD equation

+ Locally optimal expansion (with exact solves)

+ Asymptotic quadratic convergence possible(with exact solves)

+ Well-conditioned (when λ is simple)

+ Fast convergence with moderate accurate solves

+ Preconditioners can be exploited

Additional costs per step

Page 33: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 34: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

For ease of discussion . . .

Recall that harmonic Ritz vectors are better suited for se-

lecting approximate eigenpairs than Ritz vectors. In our

discussion below, we refer to Ritz pairs, but the discussion

can be extended to harmonic Ritz pairs.

For stability we need a well-conditioned bases of the search

subspace. We assume the basis v1, . . . ,vk to be orthonor-

mal and the expansion vector t is to be orthonormalised

against v1, . . . ,vk to obtain the next basis vector vk+1.

However, it may be more efficient to have orthonormality

with respect to other non-standard inner product (as the

A-, or M-inner product, . . . ) or to have bi-orthogonality.

Page 35: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),

i.e., uj ∈ Vk and ϑj ∈ C such that

Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)

or, equivalently, with Hk ≡ V∗kAVk,

uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).

Note the change in notation:we now suppress the iteration index k, while before we suppressed theordering index j in the set of all Ritz pairs at iteration step k:

including the iteration index k as well in the present notation, the Ritz

pairs (ϑk,uk) that we considered before would correspond to (ϑ(k)j ,u

(k)j )

for some j.

Page 36: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),

i.e., uj ∈ Vk and ϑj ∈ C such that

Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)

or, equivalently, with Hk ≡ V∗kAVk,

uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).

The ℓ most promising Ritz vectors u1, . . . ,uℓ are the ones

with Ritz value that ‘best’ have the property that we want

our wanted eigenvalue of A to have.

Examples. • Re(ϑj) ≥ Re(ϑj+1) (j = 1, . . . , k − 1)

if the eigenvalue of A with largest real part is wanted.

• |ϑj − τ | ≤ |ϑj+1 − τ | (j = 1, . . . , k − 1)

if the eigenvalue of A closest to some target τ ∈ C is wanted.

Page 37: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).

For stability, we rather compute the Schur decomposition

Hk = USU∗

of Hk rather than the eigenvector decomposition:

here U is k × k unitary and S is k × k upper triangular.

The Ritz values are the diagonal entries of S: ϑj = Sjj.

If Szj = ϑj zj then yj = U zj.

Theorem. The first ℓ columns of U form an orthonormal

basis of the space spanned by the eigenvectors y1, . . . , yℓ of

Hk with eigenvalues the top ℓ diagonal entries S11, . . . , Sℓℓ

of S.

Page 38: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).

For stability, we rather compute the Schur decomposition

Hk = USU∗

of Hk rather than the eigenvector decomposition:

here U is k × k unitary and S is k × k upper triangular.

A Schur decomposition can be re-ordered (using unitary

transforms, Givens rotations) such that the most promising

Ritz values are the top diagonal entries of ‘new’ S:

Theorem. There is a Schur decomposition Hk = USU∗ of

Hk such that the diagonal elements of S appear in prescri-

bed order.

Assumption. u1, . . . ,uℓ are the most promising Ritz vec-

tors and span(u1, . . . ,uℓ) = span(VkUe1, . . . ,VkUeℓ).

Page 39: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Restart

If dim(V) = k is too high, k = kmax, then we have to

restart to limit

• high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),

• high computational costs (to orthonormalise t against Vk).

Simple restart. Take V = span(u1), i.e., V1 = [u1].

Thick restart. Take V = span(u1, . . . ,uℓ) with ℓ = kmin

(i.e., Vℓ = Vk U(:,1 : ℓ) ).

Why thick restart? i.e., why ℓ = kmin > 1?

• to (partially) maintain super linear convergence

• to maintain a space that provides a good initial guess whenthe search to a next eigenpair is started.

Explanation. The second Ritz pair is likely to converge to the secondeigenpair (recall the convergence proof of the power method). Themain component of the error in u1 is probably in the direction of u2,etc.

Page 40: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Restart

If dim(V) = k is too high, k = kmax, then we have to

restart to limit

• high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),

• high computational costs (to orthonormalise t against Vk).

Simple restart. Take V = span(u1), i.e., V1 = [u1].

Thick restart. Take V = span(u1, . . . ,uℓ) with ℓ = kmin

(i.e., Vℓ = Vk U(:,1 : ℓ) ).

Example. For the case where A∗ = A.

Locally Optimal CG takes kmax = 3 and kmin = 2 and

restarts with the two dimensional space spanned by the

best Ritz vector and the best Ritz vector from the prece-

ding step.LOCG expands with a Davidson step, solve t from Mt = r.

Page 41: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Restarting Arnoldi

Arnoldi’s method requires a start with an Arnoldi relation.

If, with η ≡ hk+1,k,

AVk = Vk+1Hk = VkHk + ηvk+1e∗k,

then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,

AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)

The ℓ-vector q will not be a multiple of eℓ and (∗) is not

an Arnoldi relation.

Page 42: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Restarting Arnoldi

Arnoldi’s method requires a start with an Arnoldi relation.

If, with η ≡ hk+1,k,

AVk = Vk+1Hk = VkHk + ηvk+1e∗k,

then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,

AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)

The ℓ-vector q will not be a multiple of eℓ and (∗) is not

an Arnoldi relation.

Theorem. There is a unitary matrix Q (product of Hou-

seholder reflections) such that[Q 00∗ 1

]∗ [S

ηq∗

]Q is (ℓ+1)× ℓ upper Hessenberg.

Note. Except for vector updates Vk(U(:,1 : ℓ)Q) this allowsa restart at the cost of only low dimensional operations.

Page 43: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Implicitly Restarted Arnoldi Method

More elegant and more efficient and stable implementation

to compute U(:,1 : ℓ)Q by Sorenson is based on idea QR-

algorithm.

Page 44: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Implicitly Restarted Arnoldi Method

Select shifts µk, µk−1 . . . , µℓ+1 in C. Let p be the polynomial

p(ζ) ≡ (ζ − µk) · . . . · (ζ − µℓ+1) (ζ ∈ C).

Let v1 ≡ p(A)v1. Consider the Arnoldi relation

AVℓ = Vℓ+1 Hℓ with Vℓe1 = v1.

Theorem. This Arnoldi relation can be obtained by ap-

plying k − ℓ steps of the shifted QR-algorithm to Hk with

shifts µj finding HkU = UHk and then use the first ℓ co-

lumns of U to form Vℓ = VkU(:,1 : ℓ).

IRAM selects µj = ϑj (the less best Ritz values). This

gives the desired Arnoldi: the columns of Vℓ form an or-

thonormal basis of the space spanned by u1, . . . ,uℓ. The

eigenvalues of Hℓ are precisely ϑ1, . . . , ϑℓ.

Page 45: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Implicitly Restarted Arnoldi Method

eigs.m in Matlab is IRAM (with deflation)

eigs.m is based on ARPACK, a collection of FORTRAN

routines that implements IRAM

Lehoucq, Sorensen and Yang 1998

Page 46: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Page 47: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

Page 48: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

1) Continue that process, but, at restart, do not select λ.

Page 49: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

1) Continue that process, but, at restart, do not select λ.

Very unstable: the computed eigenpair will be computed

again, which is a waste of computational effort.

Page 50: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

1) Continue that process, but, at restart, do not select λ.

2) Deflation: remove the detected eigenvector x from the

eigenvalue problem (and, in each step, from the computa-

tional process).

Page 51: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Deflation

For stability reason, aim for Schur decomposition

AQ = QS

with Q unitary and S upper triangular.

Suppose partial Schur decomposition is available, i.e.,

AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.

Note. S11y = λy, then Ax = λx if x ≡Q1y.

Page 52: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Deflation

For stability reason, aim for Schur decomposition

AQ = QS

with Q unitary and S upper triangular.

Suppose partial Schur decomposition is available, i.e.,

AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.

Note. S11y = λy, then Ax = λx if x ≡Q1y.

Theorem. q = qℓ+1 with λ = sℓ+1,ℓ+1 satisfies

(I−Q1Q∗1)A(I−Q1Q

∗1)q = λq

Page 53: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Deflation

Deflation can be incorporated in two ways.

• Either by keeping the detected Schur vectors in the search

subspace and working with A. This approach fits Arnoldi’s

method well; see next transparency.

• Or by removing these vectors from the search subspace

and working with the deflated matrix. This approach fits

Jacobi–Davidson well.

In JD, it leads to the following JD correction equation:

(I− [Q1,u][Q∗1,u])(A− ϑI)(I− [Q1,u][Q1,u]

∗)t = −r.

Recall that enhancing the conditioning is one of the be-

nefits of including a projection in the ‘Shift and Invert’

equation. Including Q1 in the projections amplifies this ef-

fect. Note that the solution t is orthogonal to Q1 and the

search subspace gets automatically expanded with vectors

⊥Q1.

Page 54: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Deflating Arnoldi’s relation

We want to retain an Arnoldi relation at deflation.

Theorem. If AVk = Vk+1Hk, then the following properties

are equivalent

• One of the Ritz vectors uj is an eigenvector.

• A sub-diagonal element, say hℓ+1,ℓ, of Hk is 0.

• span(Vℓ) is A-invariant (whence, spanned by eigenvectors).

Idea 1. Replace small sub-diagonal elements of Hk, say

|hℓ+1,ℓ| ≤ ǫ, by 0.

Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,

use V(:, [ℓ+1 : k]) for selection (restart).

Note. span(Vℓ) = span(Q1). If Vℓ is in the Arnoldi

basis then orthonormalising Avk against Vk is equivalent

to orthonormalising (I − Q1Q∗1)A(I − Q1Q

∗1)vk against

V(:, [ℓ+1 : k]).

Page 55: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Deflating Arnoldi’s relation

We want to retain an Arnoldi relation at deflation.

Theorem. If AVk = Vk+1Hk, then the following properties

are equivalent

• One of the Ritz vectors uj is an eigenvector.

• A sub-diagonal element, say hℓ+1,ℓ, of Hk is 0.

• span(Vℓ) is A-invariant (whence, spanned by eigenvectors).

Idea 1. Replace small sub-diagonal elements of Hk, say

|hℓ+1,ℓ| ≤ ǫ, by 0.

Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,

use V(:, [ℓ+1 : k]) for selection (restart).

Note. Idea 1 is unstable and needs stabilisation.

Page 56: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Example: JD + options

The next transparencies illustrate Jacobi-Davidson with

thick restart and deflation

Page 57: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

1. Start: Choose an initial z.

• Set Λ = [ ], U = [ ], V = [ ].

2. Repeat:

a. V ← ModGS([V |z ]), W ← AV

b. Compute H ← V∗W.

c. Compute H = QRQ∗

with Q∗Q = I, R upper ∆,

|R11 − τ | ≤ |R22 − τ | ≤ . . ..

d. ϑ ← R11, y ← Qe1, u ← Vy ,

r ← ModGSU(Au− ϑu).

e. If ‖ r ‖ ≤ tol

• Λ ← [ Λ |ϑ ], U ← [U |u ].

• J=[ e2 | . . . | edim(V) ],

V← V(QJ), Q ← I, R← J∗RJ.

• Goto 2.d.

f . Solve (app.) z ⊥ U ≡ [U |u ]

(I− U U∗) (A− ϑ I) (I− U U

∗) z = −r.

Page 58: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

-10

-5

0 10 20 30 40 50 60

log

10

of

resid

ua

l n

orm

number of flops x 1e6

GMRES Ritz1

-10

-5

log

10

of

resid

ua

l n

orm

GMRES Harmonic1

max(dim(span(V))) = 15, after restart dim(V) = 10

Page 59: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Spectrum MHD test problem

−600

−450

−300

−150

0

150

300

450

600

−50 −40 −30 −20 −10 0 10

C

Entire spectrum of the MHD test problem (n = 416)

showing the correct location ( ) of the eigenvalues.

Page 60: Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A −τI)−1uk. (s&i) Linear convergence towards eigenvector with eigenvalue closest to

Alfven spectrum

0.0

0.2

0.4

0.6

0.8

1.0

−1.0 −0.8 −0.6 −0.4 −0.2 0.0

C

∗�

The MHD test problem (n = 416).

Correct eigenvalues ( ◦ ), target ( ∗� ).