Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A...
Embed Size (px)
Transcript of Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A...

Uppsala, April 2014
Subspace methods
Gerard SleijpenDepartment of Mathematics
http://www.staff.science.uu.nl/∼sleij101/

Ax = b Ax = λx
Subspace methods
Iterate until sufficiently accurate:
• Expansion. Expand the search subspace Vk.Restart if dim(Vk) is too large.
• Extraction. Extract an appropriate approximatesolution (ϑ,u) from the search subspace.
Practical aspects.
Construct basis v1, . . . ,vk of the search subspace Vk.
Notation. Vk = [v1, . . . , vk]; Vk = span(Vk).
• Compute expansion vector t, “orthogonalise” to vk+1.orthogonalize, Aorthogonalize, biorthogonalize, . . .
• Approximate solution uk ≡ Vk yk; compute yk.

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

Krylov subspace expansion
The columns of Vk = [v1, . . . ,vk] form a (orthonormal)
Krylov basis: then t = Avk is an expansion vector.
Examples. For Ax = λx.
• Arnoldi ((harmonic) Ritz extraction).
Lanczos in case A is Hermitian.

Krylov subspace expansion
The columns of Vk = [v1, . . . ,vk] form a (orthonormal)
Krylov basis: then t = Avk is an expansion vector.
Examples. For Ax = λx.
• Arnoldi ((harmonic) Ritz extraction).
Lanczos in case A is Hermitian.
• Shift and Invert Arnoldi:
t = (A− τ I)−1vk to generate a basis
for the search subspace Kk((A− τ I)−1,v1).
Shift and Invert Lanczos for the Hermitian case

Krylov subspace expansion
The columns of Vk = [v1, . . . ,vk] form a (orthonormal)
Krylov basis: then t = Avk is an expansion vector.
Examples. For Ax = λx.
• Arnoldi ((harmonic) Ritz extraction).
Lanczos in case A is Hermitian.
• Shift and Invert Arnoldi:
t = (A− τ I)−1vk to generate a basis
for the search subspace Kk((A− τ I)−1,v1).
SI expansion amplifies components of eigenvectors with
eigenvalue close to the target τ .

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

Convergence without subspace acceleration
Shift & Invert uk+1 = (A− τ I)−1uk. (s&i)
Linear convergence towards eigenvector with eigenvalue
closest to τ .
Rayleigh Quotient Iteration
uk+1 = (A− ρk I)−1uk, where ρk ≡ ρ(uk).
Asymptotic quadratic convergence towards an eigenvector.
Asymptotic cubic convergence with twosided Rayleigh quo
tient iteration, in particular, if A Hermitian.

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

Accelerated RQI. Expand with
t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)
and u approximate eigenvector in span(V).

Accelerated RQI. Expand with
t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)
and u approximate eigenvector in span(V).
+ Fast convergence (RQI)
+ Subspace allows better control of the process.
Search subspace is not a Krylov subspace.

Accelerated RQI. Expand with
t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)
and u approximate eigenvector in span(V).
+ Fast convergence (RQI)
+ Subspace allows better control of the process.
Search subspace is not a Krylov subspace.
[Ruhe 82]Rational Krylov Sequence method allows
different shifts in Shift and Invert (generating a subspace of
rational functions in A times u0, rather than polynomials),
yet it works with Hessenberg matrices (making computa
tions a little more efficient).

Accelerated RQI. Expand with
t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)
and u approximate eigenvector in span(V).
+ Fast convergence (RQI)
+ Subspace allows better control of the process.
Search subspace is not a Krylov subspace.
[Ruhe 82]Rational Krylov Sequence method allows
efficient computations.
Computational costs
S& I: Solving (A− τ I)uk = uk+1 requires one LUdec. in total
RQI: Solving (A− ρkI)uk = uk+1 requires one LUdec. each step
May not be feasible if n large.

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

Ax = λx.
If (ϑ,u) is an approximate eigenpair, r ≡ Au− ϑu
(e.g., ϑ = ρ(u)), then the solution t ⊥ u of
(I− uu∗)(A− ϑI)(I− uu∗)t = −r
is the (first order wrt ‖r‖2) correction of u.
[Sleijpen vd Vorst 95] JacobiDavidson

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
The JacobiDavidson correction equation (jd) can be
viewed as a Newton correction equation for the nonlinear
problem
F(λ, x) ≡ (1− ‖x‖22,Ax− λx) = (0,0) :
(jd) can be related to the Jacobian of F : Cn+1 → Cn+1.

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Note. If ϑ ≈ λ,
then the system (A− ϑI)t = u is ill conditioned.
Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system
(jd) as a system in the space u⊥ is wellconditioned.
Krylov subspace solvers with initial guess x0 = 0 are suit
able for solving in u⊥.
Better conditioning
• faster convergence of the linear solver(deflation of small eigenvalue),
• more stability.

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Note. If ϑ ≈ λ,
then the system (A− ϑI)t = u is ill conditioned.
Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system
(jd) as a system in the space u⊥ is wellconditioned.
There are blockversions to deal with the nonsimple eigen
value case (but the simple eigenvalue version often works
well in the nonsimple case).
Block versions require to know the multiplicity in advance.
Block versions are attractive for computational reasons
(exploit parallelism).

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Solve approximately with preconditioned iterative linear sol
ver as GMRES (or MINRES of A∗ = A).
Often a fixed modest number of steps already lead to fast
convergence.
Issue. How many steps are optimal?
Many steps in the inner loop (to solve (jd))
high quality search subspace of low dimension.
Optimal strategy is available in case A = A∗.

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ Mt = −r+ βu with β st t ⊥ u
⇔ t = −M−1r+ βM−1u ⊥ u ⇒ β = u∗M−1r
u∗M−1u
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u
Expansion by t = −M−1r (d) Davidson ’75
Expansion by t = −(I− wu∗
u∗w)M−1r (o) Olsen ’93

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u
Interpretation. If M = A− ϑI, then u = M−1r. Hence,
effective expansion with w = M−1u: RQI expansion.

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u
Interpretation. If M = A− ϑI, then u = M−1r. Hence,
effective expansion with w = M−1u: RQI expansion.
+ expansion equation (jd) better conditioned then (rqi).

0 5 10 15 20 25 30 35 4010
−10
10−8
10−6
10−4
10−2
100
102
norm
of JD
resid
ual
number of outer iterations
5 steps GMRES
− Jacobi−Davidson
−. Gen. Davidson
... Inexact Shift & Invert
A is SHERMAN4 finding the eigenvalue closest to τ = 0.5
(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

0 5 10 15 20 25 30 35 4010
−12
10−10
10−8
10−6
10−4
10−2
100
102
norm
of JD
resid
ual
number of outer iterations
10 steps GMRES
− Jacobi−Davidson
−. Gen. Davidson
... Inexact Shift & Invert
A is SHERMAN4 finding the eigenvalue closest to τ = 0.5
(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

0 5 10 15 20 25 30 35 4010
−14
10−12
10−10
10−8
10−6
10−4
10−2
100
102
norm
of JD
resid
ual
number of outer iterations
25 steps GMRES
− Jacobi−Davidson
−. Gen. Davidson
... Inexact Shift & Invert
A is SHERMAN4 finding the eigenvalue closest to τ = 0.5
(which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u
Precondition (jd) by (I− wu∗
u∗w)M−1, where w ≡M−1u
With x0 = 0, preconditioned Krylov requires mult. by
(I− wu∗
u∗w)M−1(A− ϑI)
with righthand side vector (I− wu∗
u∗w)M−1r.

(Approximately) solving for t ⊥ u
(I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)
If solved exactly ⇒ asymptotic quadratic convergence
Approximate solves. M ≈ A− ϑI
t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r
⇔ t = −(I− wu∗
u∗w)M−1r, where w ≡M−1u
Precondition (jd) by (I− wu∗
u∗w)M−1, where w ≡M−1u
With x0 = 0, preconditioned Krylov requires mult. by
(I− wu∗
u∗w)M−1(A− ϑI)
with righthand side vector (I− wu∗
u∗w)M−1r.
Additional costs. (additional to RQI)
per step mult. by I− wu∗
u∗w : 1 AXPY, 1 DOT per step
per Krylov run: 1 solve of Mw = u.

Jacobi–Davidson
• Subspace method
+ Accelerated convergence
+ Steering possibilities
+ variety of selection methods
More costly steps
• Expansion vectors from JD equation
+ Locally optimal expansion (with exact solves)
+ Asymptotic quadratic convergence possible(with exact solves)
+ Wellconditioned (when λ is simple)
+ Fast convergence with moderate accurate solves
+ Preconditioners can be exploited
Additional costs per step

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

For ease of discussion . . .
Recall that harmonic Ritz vectors are better suited for se
lecting approximate eigenpairs than Ritz vectors. In our
discussion below, we refer to Ritz pairs, but the discussion
can be extended to harmonic Ritz pairs.
For stability we need a wellconditioned bases of the search
subspace. We assume the basis v1, . . . ,vk to be orthonor
mal and the expansion vector t is to be orthonormalised
against v1, . . . ,vk to obtain the next basis vector vk+1.
However, it may be more efficient to have orthonormality
with respect to other nonstandard inner product (as the
A, or Minner product, . . . ) or to have biorthogonality.

The most promising Ritz vectors
If Vk = [v1, . . . , vk] spans the search subspace Vk,
then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),
i.e., uj ∈ Vk and ϑj ∈ C such that
Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)
or, equivalently, with Hk ≡ V∗kAVk,
uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).
Note the change in notation:we now suppress the iteration index k, while before we suppressed theordering index j in the set of all Ritz pairs at iteration step k:
including the iteration index k as well in the present notation, the Ritz
pairs (ϑk,uk) that we considered before would correspond to (ϑ(k)j ,u
(k)j )
for some j.

The most promising Ritz vectors
If Vk = [v1, . . . , vk] spans the search subspace Vk,
then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),
i.e., uj ∈ Vk and ϑj ∈ C such that
Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)
or, equivalently, with Hk ≡ V∗kAVk,
uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).
The ℓ most promising Ritz vectors u1, . . . ,uℓ are the ones
with Ritz value that ‘best’ have the property that we want
our wanted eigenvalue of A to have.
Examples. • Re(ϑj) ≥ Re(ϑj+1) (j = 1, . . . , k − 1)
if the eigenvalue of A with largest real part is wanted.
• ϑj − τ  ≤ ϑj+1 − τ  (j = 1, . . . , k − 1)
if the eigenvalue of A closest to some target τ ∈ C is wanted.

The most promising Ritz vectors
If Vk = [v1, . . . , vk] spans the search subspace Vk,
then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).
For stability, we rather compute the Schur decomposition
Hk = USU∗
of Hk rather than the eigenvector decomposition:
here U is k × k unitary and S is k × k upper triangular.
The Ritz values are the diagonal entries of S: ϑj = Sjj.
If Szj = ϑj zj then yj = U zj.
Theorem. The first ℓ columns of U form an orthonormal
basis of the space spanned by the eigenvectors y1, . . . , yℓ of
Hk with eigenvalues the top ℓ diagonal entries S11, . . . , Sℓℓof S.

The most promising Ritz vectors
If Vk = [v1, . . . , vk] spans the search subspace Vk,
then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).
For stability, we rather compute the Schur decomposition
Hk = USU∗
of Hk rather than the eigenvector decomposition:
here U is k × k unitary and S is k × k upper triangular.
A Schur decomposition can be reordered (using unitary
transforms, Givens rotations) such that the most promising
Ritz values are the top diagonal entries of ‘new’ S:
Theorem. There is a Schur decomposition Hk = USU∗ of
Hk such that the diagonal elements of S appear in prescri
bed order.
Assumption. u1, . . . ,uℓ are the most promising Ritz vec
tors and span(u1, . . . ,uℓ) = span(VkUe1, . . . ,VkUeℓ).

Restart
If dim(V) = k is too high, k = kmax, then we have to
restart to limit
• high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),
• high computational costs (to orthonormalise t against Vk).
Simple restart. Take Ṽ = span(u1), i.e., Ṽ1 = [u1].
Thick restart. Take Ṽ = span(u1, . . . ,uℓ) with ℓ = kmin
(i.e., Ṽℓ = Vk U(:,1 : ℓ) ).
Why thick restart? i.e., why ℓ = kmin > 1?
• to (partially) maintain super linear convergence
• to maintain a space that provides a good initial guess whenthe search to a next eigenpair is started.
Explanation. The second Ritz pair is likely to converge to the secondeigenpair (recall the convergence proof of the power method). Themain component of the error in u1 is probably in the direction of u2,etc.

Restart
If dim(V) = k is too high, k = kmax, then we have to
restart to limit
• high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),
• high computational costs (to orthonormalise t against Vk).
Simple restart. Take Ṽ = span(u1), i.e., Ṽ1 = [u1].
Thick restart. Take Ṽ = span(u1, . . . ,uℓ) with ℓ = kmin
(i.e., Ṽℓ = Vk U(:,1 : ℓ) ).
Example. For the case where A∗ = A.
Locally Optimal CG takes kmax = 3 and kmin = 2 and
restarts with the two dimensional space spanned by the
best Ritz vector and the best Ritz vector from the prece
ding step.LOCG expands with a Davidson step, solve t from Mt = r.

Restarting Arnoldi
Arnoldi’s method requires a start with an Arnoldi relation.
If, with η ≡ hk+1,k,
AVk = Vk+1Hk = VkHk + ηvk+1e∗k,
then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,
AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)
The ℓvector q will not be a multiple of eℓ and (∗) is not
an Arnoldi relation.

Restarting Arnoldi
Arnoldi’s method requires a start with an Arnoldi relation.
If, with η ≡ hk+1,k,
AVk = Vk+1Hk = VkHk + ηvk+1e∗k,
then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,
AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)
The ℓvector q will not be a multiple of eℓ and (∗) is not
an Arnoldi relation.
Theorem. There is a unitary matrix Q (product of Hou
seholder reflections) such that[Q 00∗ 1
]∗ [S
ηq∗
]Q is (ℓ+1)× ℓ upper Hessenberg.
Note. Except for vector updates Vk(U(:,1 : ℓ)Q) this allowsa restart at the cost of only low dimensional operations.

Implicitly Restarted Arnoldi Method
More elegant and more efficient and stable implementation
to compute U(:,1 : ℓ)Q by Sorenson is based on idea QR
algorithm.

Implicitly Restarted Arnoldi Method
Select shifts µk, µk−1 . . . , µℓ+1 in C. Let p be the polynomial
p(ζ) ≡ (ζ − µk) · . . . · (ζ − µℓ+1) (ζ ∈ C).
Let ṽ1 ≡ p(A)v1. Consider the Arnoldi relation
AṼℓ = Ṽℓ+1 H̃ℓ with Ṽℓe1 = ṽ1.
Theorem. This Arnoldi relation can be obtained by ap
plying k − ℓ steps of the shifted QRalgorithm to Hk with
shifts µj finding HkU = UH̃k and then use the first ℓ co
lumns of U to form Ṽℓ = VkU(:,1 : ℓ).
IRAM selects µj = ϑj (the less best Ritz values). This
gives the desired Arnoldi: the columns of Ṽℓ form an or
thonormal basis of the space spanned by u1, . . . ,uℓ. The
eigenvalues of H̃ℓ are precisely ϑ1, . . . , ϑℓ.

Implicitly Restarted Arnoldi Method
eigs.m in Matlab is IRAM (with deflation)
eigs.m is based on ARPACK, a collection of FORTRAN
routines that implements IRAM
Lehoucq, Sorensen and Yang 1998

Program Lecture 4
• Expansion
◦ Krylov subspace approach
Lanczos, Arnoldi, Shift and Invert Arnoldi
◦ Convergence
◦ Accelerated Rayleigh Quotient Iteration
Rational Krylov Sequence method
◦ Optimal expansion
JacobiDavidson
• Restart
LOCG, Implicitly Restarted Arnoldi Method
• Deflation

Towards the next eigenpair
If an eigenvector x with eigenvalue λ has been detected
(computed), how to continue the computational process
for the next eigenpair?

Towards the next eigenpair
If an eigenvector x with eigenvalue λ has been detected
(computed), how to continue the computational process
for the next eigenpair?
1) Continue that process, but, at restart, do not select λ.

Towards the next eigenpair
If an eigenvector x with eigenvalue λ has been detected
(computed), how to continue the computational process
for the next eigenpair?
1) Continue that process, but, at restart, do not select λ.
Very unstable: the computed eigenpair will be computed
again, which is a waste of computational effort.

Towards the next eigenpair
If an eigenvector x with eigenvalue λ has been detected
(computed), how to continue the computational process
for the next eigenpair?
1) Continue that process, but, at restart, do not select λ.
2) Deflation: remove the detected eigenvector x from the
eigenvalue problem (and, in each step, from the computa
tional process).

Deflation
For stability reason, aim for Schur decomposition
AQ = QS
with Q unitary and S upper triangular.
Suppose partial Schur decomposition is available, i.e.,
AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.
Note. S11y = λy, then Ax = λx if x ≡Q1y.

Deflation
For stability reason, aim for Schur decomposition
AQ = QS
with Q unitary and S upper triangular.
Suppose partial Schur decomposition is available, i.e.,
AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.
Note. S11y = λy, then Ax = λx if x ≡Q1y.
Theorem. q = qℓ+1 with λ = sℓ+1,ℓ+1 satisfies
(I−Q1Q∗1)A(I−Q1Q
∗1)q = λq

Deflation
Deflation can be incorporated in two ways.
• Either by keeping the detected Schur vectors in the search
subspace and working with A. This approach fits Arnoldi’s
method well; see next transparency.
• Or by removing these vectors from the search subspace
and working with the deflated matrix. This approach fits
Jacobi–Davidson well.
In JD, it leads to the following JD correction equation:
(I− [Q1,u][Q∗1,u])(A− ϑI)(I− [Q1,u][Q1,u]
∗)t = −r.
Recall that enhancing the conditioning is one of the be
nefits of including a projection in the ‘Shift and Invert’
equation. Including Q1 in the projections amplifies this ef
fect. Note that the solution t is orthogonal to Q1 and the
search subspace gets automatically expanded with vectors
⊥Q1.

Deflating Arnoldi’s relation
We want to retain an Arnoldi relation at deflation.
Theorem. If AVk = Vk+1Hk, then the following properties
are equivalent
• One of the Ritz vectors uj is an eigenvector.
• A subdiagonal element, say hℓ+1,ℓ, of Hk is 0.
• span(Vℓ) is Ainvariant (whence, spanned by eigenvectors).
Idea 1. Replace small subdiagonal elements of Hk, say
hℓ+1,ℓ ≤ ǫ, by 0.
Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,
use V(:, [ℓ+1 : k]) for selection (restart).
Note. span(Vℓ) = span(Q1). If Vℓ is in the Arnoldi
basis then orthonormalising Avk against Vk is equivalent
to orthonormalising (I − Q1Q∗1)A(I − Q1Q
∗1)vk against
V(:, [ℓ+1 : k]).

Deflating Arnoldi’s relation
We want to retain an Arnoldi relation at deflation.
Theorem. If AVk = Vk+1Hk, then the following properties
are equivalent
• One of the Ritz vectors uj is an eigenvector.
• A subdiagonal element, say hℓ+1,ℓ, of Hk is 0.
• span(Vℓ) is Ainvariant (whence, spanned by eigenvectors).
Idea 1. Replace small subdiagonal elements of Hk, say
hℓ+1,ℓ ≤ ǫ, by 0.
Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,
use V(:, [ℓ+1 : k]) for selection (restart).
Note. Idea 1 is unstable and needs stabilisation.

Example: JD + options
The next transparencies illustrate JacobiDavidson with
thick restart and deflation

1. Start: Choose an initial z.
• Set Λ = [ ], U = [ ], V = [ ].
2. Repeat:
a. V ← ModGS([V z ]), W ← AV
b. Compute H ← V∗W.
c. Compute H = QRQ∗
with Q∗Q = I, R upper ∆,
R11 − τ  ≤ R22 − τ  ≤ . . ..
d. ϑ ← R11, y ← Qe1, u ← Vy ,
r ← ModGSU(Au− ϑu).
e. If ‖ r ‖ ≤ tol
• Λ ← [ Λ ϑ ], U ← [U u ].
• J=[ e2  . . .  edim(V) ],
V← V(QJ), Q ← I, R← J∗RJ.
• Goto 2.d.
f . Solve (app.) z ⊥ Ũ ≡ [U u ]
(I− Ũ Ũ∗) (A− ϑ I) (I− Ũ Ũ
∗) z = −r.

10
5
0 10 20 30 40 50 60
log
10
of
resid
ua
l n
orm
number of flops x 1e6
GMRES Ritz1
10
5
log
10
of
resid
ua
l n
orm
GMRES Harmonic1
max(dim(span(V))) = 15, after restart dim(V) = 10

Spectrum MHD test problem
−600
−450
−300
−150
0
150
300
450
600
−50 −40 −30 −20 −10 0 10
C
Entire spectrum of the MHD test problem (n = 416)
showing the correct location ( ) of the eigenvalues.

Alfvén spectrum
0.0
0.2
0.4
0.6
0.8
1.0
−1.0 −0.8 −0.6 −0.4 −0.2 0.0
C
∗�
The MHD test problem (n = 416).
Correct eigenvalues ( ◦ ), target ( ∗� ).