Uppsala, April 2014

Subspace methods

Gerard SleijpenDepartment of Mathematics

http://www.staff.science.uu.nl/∼sleij101/

Ax = b Ax = λx

Subspace methods

Iterate until sufficiently accurate:

• Expansion. Expand the search subspace Vk.Restart if dim(Vk) is too large.

• Extraction. Extract an appropriate approximatesolution (ϑ,u) from the search subspace.

Practical aspects.

Construct basis v1, . . . ,vk of the search subspace Vk.

Notation. Vk = [v1, . . . , vk]; Vk = span(Vk).

• Compute expansion vector t, “orthogonalise” to vk+1.orthogonalize, A-orthogonalize, bi-orthogonalize, . . .

• Approximate solution uk ≡ Vk yk; compute yk.

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Krylov subspace expansion

The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

Krylov basis: then t = Avk is an expansion vector.

Examples. For Ax = λx.

• Arnoldi ((harmonic) Ritz extraction).

Lanczos in case A is Hermitian.

Krylov subspace expansion

The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

Krylov basis: then t = Avk is an expansion vector.

Examples. For Ax = λx.

• Arnoldi ((harmonic) Ritz extraction).

Lanczos in case A is Hermitian.

• Shift and Invert Arnoldi:

t = (A− τ I)−1vk to generate a basis

for the search subspace Kk((A− τ I)−1,v1).

Shift and Invert Lanczos for the Hermitian case

Krylov subspace expansion

The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

Krylov basis: then t = Avk is an expansion vector.

Examples. For Ax = λx.

• Arnoldi ((harmonic) Ritz extraction).

Lanczos in case A is Hermitian.

• Shift and Invert Arnoldi:

t = (A− τ I)−1vk to generate a basis

for the search subspace Kk((A− τ I)−1,v1).

SI expansion amplifies components of eigenvectors with

eigenvalue close to the target τ .

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Convergence without subspace acceleration

Shift & Invert uk+1 = (A− τ I)−1uk. (s&i)

Linear convergence towards eigenvector with eigenvalue

closest to τ .

Rayleigh Quotient Iteration

uk+1 = (A− ρk I)−1uk, where ρk ≡ ρ(uk).

Asymptotic quadratic convergence towards an eigenvector.

Asymptotic cubic convergence with two-sided Rayleigh quo-

tient iteration, in particular, if A Hermitian.

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

+ Fast convergence (RQI)

+ Subspace allows better control of the process.

Search subspace is not a Krylov subspace.

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

+ Fast convergence (RQI)

+ Subspace allows better control of the process.

Search subspace is not a Krylov subspace.

[Ruhe 82]Rational Krylov Sequence method allows

different shifts in Shift and Invert (generating a subspace of

rational functions in A times u0, rather than polynomials),

yet it works with Hessenberg matrices (making computa-

tions a little more efficient).

Accelerated RQI. Expand with

t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

and u approximate eigenvector in span(V).

+ Fast convergence (RQI)

+ Subspace allows better control of the process.

Search subspace is not a Krylov subspace.

[Ruhe 82]Rational Krylov Sequence method allows

efficient computations.

Computational costs

S& I: Solving (A− τ I)uk = uk+1 requires one LU-dec. in total

RQI: Solving (A− ρkI)uk = uk+1 requires one LU-dec. each step

May not be feasible if n large.

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Ax = λx.

If (ϑ,u) is an approximate eigenpair, r ≡ Au− ϑu

(e.g., ϑ = ρ(u)), then the solution t ⊥ u of

(I− uu∗)(A− ϑI)(I− uu∗)t = −r

is the (first order wrt ‖r‖2) correction of u.

[Sleijpen vd Vorst 95] Jacobi-Davidson

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

The Jacobi-Davidson correction equation (jd) can be

viewed as a Newton correction equation for the non-linear

problem

F(λ, x) ≡ (1− ‖x‖22,Ax− λx) = (0,0) :

(jd) can be related to the Jacobian of F : Cn+1 → Cn+1.

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Note. If ϑ ≈ λ,

then the system (A− ϑI)t = u is ill conditioned.

Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system

(jd) as a system in the space u⊥ is well-conditioned.

Krylov subspace solvers with initial guess x0 = 0 are suit-

able for solving in u⊥.

Better conditioning

• faster convergence of the linear solver(deflation of small eigenvalue),

• more stability.

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Note. If ϑ ≈ λ,

then the system (A− ϑI)t = u is ill conditioned.

Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system

(jd) as a system in the space u⊥ is well-conditioned.

There are block-versions to deal with the non-simple eigen-

value case (but the simple eigenvalue version often works

well in the non-simple case).

Block versions require to know the multiplicity in advance.

Block versions are attractive for computational reasons

(exploit parallelism).

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Solve approximately with preconditioned iterative linear sol-

ver as GMRES (or MINRES of A∗ = A).

Often a fixed modest number of steps already lead to fast

convergence.

Issue. How many steps are optimal?

Many steps in the inner loop (to solve (jd))

high quality search subspace of low dimension.

Optimal strategy is available in case A = A∗.

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ Mt = −r+ βu with β st t ⊥ u

⇔ t = −M−1r+ βM−1u ⊥ u ⇒ β = u∗M−1ru∗M−1u

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Expansion by t = −M−1r (d) Davidson ’75

Expansion by t = −(I− wu∗u∗w)M−1r (o) Olsen ’93

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Interpretation. If M = A− ϑI, then u = M−1r. Hence,

effective expansion with w = M−1u: RQI expansion.

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Interpretation. If M = A− ϑI, then u = M−1r. Hence,

effective expansion with w = M−1u: RQI expansion.

+ expansion equation (jd) better conditioned then (rqi).

0 5 10 15 20 25 30 35 4010

−10

10−8

10−6

10−4

10−2

100

102

norm

of JD

resid

ual

number of outer iterations

5 steps GMRES

− Jacobi−Davidson

−. Gen. Davidson

... Inexact Shift & Invert

A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

0 5 10 15 20 25 30 35 4010

−12

10−10

10−8

10−6

10−4

10−2

100

102

norm

of JD

resid

ual

number of outer iterations

10 steps GMRES

− Jacobi−Davidson

−. Gen. Davidson

... Inexact Shift & Invert

A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

0 5 10 15 20 25 30 35 4010

−14

10−12

10−10

10−8

10−6

10−4

10−2

100

102

norm

of JD

resid

ual

number of outer iterations

25 steps GMRES

− Jacobi−Davidson

−. Gen. Davidson

... Inexact Shift & Invert

A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Precondition (jd) by (I− wu∗u∗w)M−1, where w ≡M−1u

With x0 = 0, preconditioned Krylov requires mult. by

(I− wu∗u∗w)M−1(A− ϑI)

with right-hand side vector (I− wu∗u∗w)M−1r.

(Approximately) solving for t ⊥ u

(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

If solved exactly ⇒ asymptotic quadratic convergence

Approximate solves. M ≈ A− ϑI

t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

⇔ t = −(I− wu∗u∗w)M−1r, where w ≡M−1u

Precondition (jd) by (I− wu∗u∗w)M−1, where w ≡M−1u

With x0 = 0, preconditioned Krylov requires mult. by

(I− wu∗u∗w)M−1(A− ϑI)

with right-hand side vector (I− wu∗u∗w)M−1r.

Additional costs. (additional to RQI)

per step mult. by I− wu∗u∗w : 1 AXPY, 1 DOT per step

per Krylov run: 1 solve of Mw = u.

Jacobi–Davidson

• Subspace method

+ Accelerated convergence

+ Steering possibilities

+ variety of selection methods

More costly steps

• Expansion vectors from JD equation

+ Locally optimal expansion (with exact solves)

+ Asymptotic quadratic convergence possible(with exact solves)

+ Well-conditioned (when λ is simple)

+ Fast convergence with moderate accurate solves

+ Preconditioners can be exploited

Additional costs per step

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

For ease of discussion . . .

Recall that harmonic Ritz vectors are better suited for se-

lecting approximate eigenpairs than Ritz vectors. In our

discussion below, we refer to Ritz pairs, but the discussion

can be extended to harmonic Ritz pairs.

For stability we need a well-conditioned bases of the search

subspace. We assume the basis v1, . . . ,vk to be orthonor-

mal and the expansion vector t is to be orthonormalised

against v1, . . . ,vk to obtain the next basis vector vk+1.

However, it may be more efficient to have orthonormality

with respect to other non-standard inner product (as the

A-, or M-inner product, . . . ) or to have bi-orthogonality.

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),

i.e., uj ∈ Vk and ϑj ∈ C such that

Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)

or, equivalently, with Hk ≡ V∗kAVk,

uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).

Note the change in notation:we now suppress the iteration index k, while before we suppressed theordering index j in the set of all Ritz pairs at iteration step k:

including the iteration index k as well in the present notation, the Ritz

pairs (ϑk,uk) that we considered before would correspond to (ϑ(k)j ,u

(k)j )

for some j.

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),

i.e., uj ∈ Vk and ϑj ∈ C such that

Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)

or, equivalently, with Hk ≡ V∗kAVk,

uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).

The ℓ most promising Ritz vectors u1, . . . ,uℓ are the ones

with Ritz value that ‘best’ have the property that we want

our wanted eigenvalue of A to have.

Examples. • Re(ϑj) ≥ Re(ϑj+1) (j = 1, . . . , k − 1)

if the eigenvalue of A with largest real part is wanted.

• |ϑj − τ | ≤ |ϑj+1 − τ | (j = 1, . . . , k − 1)

if the eigenvalue of A closest to some target τ ∈ C is wanted.

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).

For stability, we rather compute the Schur decomposition

Hk = USU∗

of Hk rather than the eigenvector decomposition:

here U is k × k unitary and S is k × k upper triangular.

The Ritz values are the diagonal entries of S: ϑj = Sjj.

If Szj = ϑj zj then yj = U zj.

Theorem. The first ℓ columns of U form an orthonormal

basis of the space spanned by the eigenvectors y1, . . . , yℓ of

Hk with eigenvalues the top ℓ diagonal entries S11, . . . , Sℓℓ

of S.

The most promising Ritz vectors

If Vk = [v1, . . . , vk] spans the search subspace Vk,

then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).

For stability, we rather compute the Schur decomposition

Hk = USU∗

of Hk rather than the eigenvector decomposition:

here U is k × k unitary and S is k × k upper triangular.

A Schur decomposition can be re-ordered (using unitary

transforms, Givens rotations) such that the most promising

Ritz values are the top diagonal entries of ‘new’ S:

Theorem. There is a Schur decomposition Hk = USU∗ of

Hk such that the diagonal elements of S appear in prescri-

bed order.

Assumption. u1, . . . ,uℓ are the most promising Ritz vec-

tors and span(u1, . . . ,uℓ) = span(VkUe1, . . . ,VkUeℓ).

Restart

If dim(V) = k is too high, k = kmax, then we have to

restart to limit

• high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),

• high computational costs (to orthonormalise t against Vk).

Simple restart. Take V = span(u1), i.e., V1 = [u1].

Thick restart. Take V = span(u1, . . . ,uℓ) with ℓ = kmin

(i.e., Vℓ = Vk U(:,1 : ℓ) ).

Why thick restart? i.e., why ℓ = kmin > 1?

• to (partially) maintain super linear convergence

• to maintain a space that provides a good initial guess whenthe search to a next eigenpair is started.

Explanation. The second Ritz pair is likely to converge to the secondeigenpair (recall the convergence proof of the power method). Themain component of the error in u1 is probably in the direction of u2,etc.

Restart

If dim(V) = k is too high, k = kmax, then we have to

restart to limit

• high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),

• high computational costs (to orthonormalise t against Vk).

Simple restart. Take V = span(u1), i.e., V1 = [u1].

Thick restart. Take V = span(u1, . . . ,uℓ) with ℓ = kmin

(i.e., Vℓ = Vk U(:,1 : ℓ) ).

Example. For the case where A∗ = A.

Locally Optimal CG takes kmax = 3 and kmin = 2 and

restarts with the two dimensional space spanned by the

best Ritz vector and the best Ritz vector from the prece-

ding step.LOCG expands with a Davidson step, solve t from Mt = r.

Restarting Arnoldi

Arnoldi’s method requires a start with an Arnoldi relation.

If, with η ≡ hk+1,k,

AVk = Vk+1Hk = VkHk + ηvk+1e∗k,

then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,

AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)

The ℓ-vector q will not be a multiple of eℓ and (∗) is not

an Arnoldi relation.

Restarting Arnoldi

Arnoldi’s method requires a start with an Arnoldi relation.

If, with η ≡ hk+1,k,

AVk = Vk+1Hk = VkHk + ηvk+1e∗k,

then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,

AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)

The ℓ-vector q will not be a multiple of eℓ and (∗) is not

an Arnoldi relation.

Theorem. There is a unitary matrix Q (product of Hou-

seholder reflections) such that[Q 00∗ 1

]∗ [S

ηq∗

]Q is (ℓ+1)× ℓ upper Hessenberg.

Note. Except for vector updates Vk(U(:,1 : ℓ)Q) this allowsa restart at the cost of only low dimensional operations.

Implicitly Restarted Arnoldi Method

More elegant and more efficient and stable implementation

to compute U(:,1 : ℓ)Q by Sorenson is based on idea QR-

algorithm.

Implicitly Restarted Arnoldi Method

Select shifts µk, µk−1 . . . , µℓ+1 in C. Let p be the polynomial

p(ζ) ≡ (ζ − µk) · . . . · (ζ − µℓ+1) (ζ ∈ C).

Let v1 ≡ p(A)v1. Consider the Arnoldi relation

AVℓ = Vℓ+1 Hℓ with Vℓe1 = v1.

Theorem. This Arnoldi relation can be obtained by ap-

plying k − ℓ steps of the shifted QR-algorithm to Hk with

shifts µj finding HkU = UHk and then use the first ℓ co-

lumns of U to form Vℓ = VkU(:,1 : ℓ).

IRAM selects µj = ϑj (the less best Ritz values). This

gives the desired Arnoldi: the columns of Vℓ form an or-

thonormal basis of the space spanned by u1, . . . ,uℓ. The

eigenvalues of Hℓ are precisely ϑ1, . . . , ϑℓ.

Implicitly Restarted Arnoldi Method

eigs.m in Matlab is IRAM (with deflation)

eigs.m is based on ARPACK, a collection of FORTRAN

routines that implements IRAM

Lehoucq, Sorensen and Yang 1998

Program Lecture 4

• Expansion

◦ Krylov subspace approach

Lanczos, Arnoldi, Shift and Invert Arnoldi

◦ Convergence

◦ Accelerated Rayleigh Quotient Iteration

Rational Krylov Sequence method

◦ Optimal expansion

Jacobi-Davidson

• Restart

LOCG, Implicitly Restarted Arnoldi Method

• Deflation

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

1) Continue that process, but, at restart, do not select λ.

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

1) Continue that process, but, at restart, do not select λ.

Very unstable: the computed eigenpair will be computed

again, which is a waste of computational effort.

Towards the next eigenpair

If an eigenvector x with eigenvalue λ has been detected

(computed), how to continue the computational process

for the next eigenpair?

1) Continue that process, but, at restart, do not select λ.

2) Deflation: remove the detected eigenvector x from the

eigenvalue problem (and, in each step, from the computa-

tional process).

Deflation

For stability reason, aim for Schur decomposition

AQ = QS

with Q unitary and S upper triangular.

Suppose partial Schur decomposition is available, i.e.,

AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.

Note. S11y = λy, then Ax = λx if x ≡Q1y.

Deflation

For stability reason, aim for Schur decomposition

AQ = QS

with Q unitary and S upper triangular.

Suppose partial Schur decomposition is available, i.e.,

AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.

Note. S11y = λy, then Ax = λx if x ≡Q1y.

Theorem. q = qℓ+1 with λ = sℓ+1,ℓ+1 satisfies

(I−Q1Q∗1)A(I−Q1Q

∗1)q = λq

Deflation

Deflation can be incorporated in two ways.

• Either by keeping the detected Schur vectors in the search

subspace and working with A. This approach fits Arnoldi’s

method well; see next transparency.

• Or by removing these vectors from the search subspace

and working with the deflated matrix. This approach fits

Jacobi–Davidson well.

In JD, it leads to the following JD correction equation:

(I− [Q1,u][Q∗1,u])(A− ϑI)(I− [Q1,u][Q1,u]

∗)t = −r.

Recall that enhancing the conditioning is one of the be-

nefits of including a projection in the ‘Shift and Invert’

equation. Including Q1 in the projections amplifies this ef-

fect. Note that the solution t is orthogonal to Q1 and the

search subspace gets automatically expanded with vectors

⊥Q1.

Deflating Arnoldi’s relation

We want to retain an Arnoldi relation at deflation.

Theorem. If AVk = Vk+1Hk, then the following properties

are equivalent

• One of the Ritz vectors uj is an eigenvector.

• A sub-diagonal element, say hℓ+1,ℓ, of Hk is 0.

• span(Vℓ) is A-invariant (whence, spanned by eigenvectors).

Idea 1. Replace small sub-diagonal elements of Hk, say

|hℓ+1,ℓ| ≤ ǫ, by 0.

Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,

use V(:, [ℓ+1 : k]) for selection (restart).

Note. span(Vℓ) = span(Q1). If Vℓ is in the Arnoldi

basis then orthonormalising Avk against Vk is equivalent

to orthonormalising (I − Q1Q∗1)A(I − Q1Q

∗1)vk against

V(:, [ℓ+1 : k]).

Deflating Arnoldi’s relation

We want to retain an Arnoldi relation at deflation.

Theorem. If AVk = Vk+1Hk, then the following properties

are equivalent

• One of the Ritz vectors uj is an eigenvector.

• A sub-diagonal element, say hℓ+1,ℓ, of Hk is 0.

• span(Vℓ) is A-invariant (whence, spanned by eigenvectors).

Idea 1. Replace small sub-diagonal elements of Hk, say

|hℓ+1,ℓ| ≤ ǫ, by 0.

Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,

use V(:, [ℓ+1 : k]) for selection (restart).

Note. Idea 1 is unstable and needs stabilisation.

Example: JD + options

The next transparencies illustrate Jacobi-Davidson with

thick restart and deflation

1. Start: Choose an initial z.

• Set Λ = [ ], U = [ ], V = [ ].

2. Repeat:

a. V ← ModGS([V |z ]), W ← AV

b. Compute H ← V∗W.

c. Compute H = QRQ∗

with Q∗Q = I, R upper ∆,

|R11 − τ | ≤ |R22 − τ | ≤ . . ..

d. ϑ ← R11, y ← Qe1, u ← Vy ,

r ← ModGSU(Au− ϑu).

e. If ‖ r ‖ ≤ tol

• Λ ← [ Λ |ϑ ], U ← [U |u ].

• J=[ e2 | . . . | edim(V) ],

V← V(QJ), Q ← I, R← J∗RJ.

• Goto 2.d.

f . Solve (app.) z ⊥ U ≡ [U |u ]

(I− U U∗) (A− ϑ I) (I− U U

∗) z = −r.

-10

-5

0 10 20 30 40 50 60

log

10

of

resid

ua

l n

orm

number of flops x 1e6

GMRES Ritz1

-10

-5

log

10

of

resid

ua

l n

orm

GMRES Harmonic1

max(dim(span(V))) = 15, after restart dim(V) = 10

Spectrum MHD test problem

−600

−450

−300

−150

0

150

300

450

600

−50 −40 −30 −20 −10 0 10

C

Entire spectrum of the MHD test problem (n = 416)

showing the correct location ( ) of the eigenvalues.

Alfven spectrum

0.0

0.2

0.4

0.6

0.8

1.0

−1.0 −0.8 −0.6 −0.4 −0.2 0.0

C

∗�

The MHD test problem (n = 416).

Correct eigenvalues ( ◦ ), target ( ∗� ).