Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A...

of 60 /60
Uppsala, April 2014 Subspace methods Gerard Sleijpen Department of Mathematics http://www.staff.science.uu.nl/sleij101/

Embed Size (px)

Transcript of Uppsala, April 2014 Subspace methodssleij101/Uppsala/Lecture4.pdf · Shift & Invert uk+1 = (A...

  • Uppsala, April 2014

    Subspace methods

    Gerard SleijpenDepartment of Mathematics

    http://www.staff.science.uu.nl/∼sleij101/

  • Ax = b Ax = λx

    Subspace methods

    Iterate until sufficiently accurate:

    • Expansion. Expand the search subspace Vk.Restart if dim(Vk) is too large.

    • Extraction. Extract an appropriate approximatesolution (ϑ,u) from the search subspace.

    Practical aspects.

    Construct basis v1, . . . ,vk of the search subspace Vk.

    Notation. Vk = [v1, . . . , vk]; Vk = span(Vk).

    • Compute expansion vector t, “orthogonalise” to vk+1.orthogonalize, A-orthogonalize, bi-orthogonalize, . . .

    • Approximate solution uk ≡ Vk yk; compute yk.

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • Krylov subspace expansion

    The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

    Krylov basis: then t = Avk is an expansion vector.

    Examples. For Ax = λx.

    • Arnoldi ((harmonic) Ritz extraction).

    Lanczos in case A is Hermitian.

  • Krylov subspace expansion

    The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

    Krylov basis: then t = Avk is an expansion vector.

    Examples. For Ax = λx.

    • Arnoldi ((harmonic) Ritz extraction).

    Lanczos in case A is Hermitian.

    • Shift and Invert Arnoldi:

    t = (A− τ I)−1vk to generate a basis

    for the search subspace Kk((A− τ I)−1,v1).

    Shift and Invert Lanczos for the Hermitian case

  • Krylov subspace expansion

    The columns of Vk = [v1, . . . ,vk] form a (orthonormal)

    Krylov basis: then t = Avk is an expansion vector.

    Examples. For Ax = λx.

    • Arnoldi ((harmonic) Ritz extraction).

    Lanczos in case A is Hermitian.

    • Shift and Invert Arnoldi:

    t = (A− τ I)−1vk to generate a basis

    for the search subspace Kk((A− τ I)−1,v1).

    SI expansion amplifies components of eigenvectors with

    eigenvalue close to the target τ .

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • Convergence without subspace acceleration

    Shift & Invert uk+1 = (A− τ I)−1uk. (s&i)

    Linear convergence towards eigenvector with eigenvalue

    closest to τ .

    Rayleigh Quotient Iteration

    uk+1 = (A− ρk I)−1uk, where ρk ≡ ρ(uk).

    Asymptotic quadratic convergence towards an eigenvector.

    Asymptotic cubic convergence with two-sided Rayleigh quo-

    tient iteration, in particular, if A Hermitian.

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • Accelerated RQI. Expand with

    t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

    and u approximate eigenvector in span(V).

  • Accelerated RQI. Expand with

    t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

    and u approximate eigenvector in span(V).

    + Fast convergence (RQI)

    + Subspace allows better control of the process.

    Search subspace is not a Krylov subspace.

  • Accelerated RQI. Expand with

    t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

    and u approximate eigenvector in span(V).

    + Fast convergence (RQI)

    + Subspace allows better control of the process.

    Search subspace is not a Krylov subspace.

    [Ruhe 82]Rational Krylov Sequence method allows

    different shifts in Shift and Invert (generating a subspace of

    rational functions in A times u0, rather than polynomials),

    yet it works with Hessenberg matrices (making computa-

    tions a little more efficient).

  • Accelerated RQI. Expand with

    t = (A− ρ(u)I)−1u, where ρ(u) = u∗Au (rqi)

    and u approximate eigenvector in span(V).

    + Fast convergence (RQI)

    + Subspace allows better control of the process.

    Search subspace is not a Krylov subspace.

    [Ruhe 82]Rational Krylov Sequence method allows

    efficient computations.

    Computational costs

    S& I: Solving (A− τ I)uk = uk+1 requires one LU-dec. in total

    RQI: Solving (A− ρkI)uk = uk+1 requires one LU-dec. each step

    May not be feasible if n large.

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • Ax = λx.

    If (ϑ,u) is an approximate eigenpair, r ≡ Au− ϑu

    (e.g., ϑ = ρ(u)), then the solution t ⊥ u of

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r

    is the (first order wrt ‖r‖2) correction of u.

    [Sleijpen vd Vorst 95] Jacobi-Davidson

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    The Jacobi-Davidson correction equation (jd) can be

    viewed as a Newton correction equation for the non-linear

    problem

    F(λ, x) ≡ (1− ‖x‖22,Ax− λx) = (0,0) :

    (jd) can be related to the Jacobian of F : Cn+1 → Cn+1.

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Note. If ϑ ≈ λ,

    then the system (A− ϑI)t = u is ill conditioned.

    Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system

    (jd) as a system in the space u⊥ is well-conditioned.

    Krylov subspace solvers with initial guess x0 = 0 are suit-

    able for solving in u⊥.

    Better conditioning

    • faster convergence of the linear solver(deflation of small eigenvalue),

    • more stability.

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Note. If ϑ ≈ λ,

    then the system (A− ϑI)t = u is ill conditioned.

    Whereas, if λ is simple and (ϑ,u) ≈ (λ, x), then the system

    (jd) as a system in the space u⊥ is well-conditioned.

    There are block-versions to deal with the non-simple eigen-

    value case (but the simple eigenvalue version often works

    well in the non-simple case).

    Block versions require to know the multiplicity in advance.

    Block versions are attractive for computational reasons

    (exploit parallelism).

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Solve approximately with preconditioned iterative linear sol-

    ver as GMRES (or MINRES of A∗ = A).

    Often a fixed modest number of steps already lead to fast

    convergence.

    Issue. How many steps are optimal?

    Many steps in the inner loop (to solve (jd))

    high quality search subspace of low dimension.

    Optimal strategy is available in case A = A∗.

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ Mt = −r+ βu with β st t ⊥ u

    ⇔ t = −M−1r+ βM−1u ⊥ u ⇒ β = u∗M−1r

    u∗M−1u

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

    Expansion by t = −M−1r (d) Davidson ’75

    Expansion by t = −(I− wu∗

    u∗w)M−1r (o) Olsen ’93

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

    Interpretation. If M = A− ϑI, then u = M−1r. Hence,

    effective expansion with w = M−1u: RQI expansion.

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

    Interpretation. If M = A− ϑI, then u = M−1r. Hence,

    effective expansion with w = M−1u: RQI expansion.

    + expansion equation (jd) better conditioned then (rqi).

  • 0 5 10 15 20 25 30 35 4010

    −10

    10−8

    10−6

    10−4

    10−2

    100

    102

    norm

    of JD

    resid

    ual

    number of outer iterations

    5 steps GMRES

    − Jacobi−Davidson

    −. Gen. Davidson

    ... Inexact Shift & Invert

    A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

    (which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

  • 0 5 10 15 20 25 30 35 4010

    −12

    10−10

    10−8

    10−6

    10−4

    10−2

    100

    102

    norm

    of JD

    resid

    ual

    number of outer iterations

    10 steps GMRES

    − Jacobi−Davidson

    −. Gen. Davidson

    ... Inexact Shift & Invert

    A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

    (which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

  • 0 5 10 15 20 25 30 35 4010

    −14

    10−12

    10−10

    10−8

    10−6

    10−4

    10−2

    100

    102

    norm

    of JD

    resid

    ual

    number of outer iterations

    25 steps GMRES

    − Jacobi−Davidson

    −. Gen. Davidson

    ... Inexact Shift & Invert

    A is SHERMAN4 finding the eigenvalue closest to τ = 0.5

    (which is the fifth smallest eigenvalue),M is ILU(2) of A− τ I

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

    Precondition (jd) by (I− wu∗

    u∗w)M−1, where w ≡M−1u

    With x0 = 0, preconditioned Krylov requires mult. by

    (I− wu∗

    u∗w)M−1(A− ϑI)

    with right-hand side vector (I− wu∗

    u∗w)M−1r.

  • (Approximately) solving for t ⊥ u

    (I− uu∗)(A− ϑI)(I− uu∗)t = −r (jd)

    If solved exactly ⇒ asymptotic quadratic convergence

    Approximate solves. M ≈ A− ϑI

    t ⊥ u such that (I− uu∗)M(I− uu∗)t = −r

    ⇔ t = −(I− wu∗

    u∗w)M−1r, where w ≡M−1u

    Precondition (jd) by (I− wu∗

    u∗w)M−1, where w ≡M−1u

    With x0 = 0, preconditioned Krylov requires mult. by

    (I− wu∗

    u∗w)M−1(A− ϑI)

    with right-hand side vector (I− wu∗

    u∗w)M−1r.

    Additional costs. (additional to RQI)

    per step mult. by I− wu∗

    u∗w : 1 AXPY, 1 DOT per step

    per Krylov run: 1 solve of Mw = u.

  • Jacobi–Davidson

    • Subspace method

    + Accelerated convergence

    + Steering possibilities

    + variety of selection methods

    More costly steps

    • Expansion vectors from JD equation

    + Locally optimal expansion (with exact solves)

    + Asymptotic quadratic convergence possible(with exact solves)

    + Well-conditioned (when λ is simple)

    + Fast convergence with moderate accurate solves

    + Preconditioners can be exploited

    Additional costs per step

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • For ease of discussion . . .

    Recall that harmonic Ritz vectors are better suited for se-

    lecting approximate eigenpairs than Ritz vectors. In our

    discussion below, we refer to Ritz pairs, but the discussion

    can be extended to harmonic Ritz pairs.

    For stability we need a well-conditioned bases of the search

    subspace. We assume the basis v1, . . . ,vk to be orthonor-

    mal and the expansion vector t is to be orthonormalised

    against v1, . . . ,vk to obtain the next basis vector vk+1.

    However, it may be more efficient to have orthonormality

    with respect to other non-standard inner product (as the

    A-, or M-inner product, . . . ) or to have bi-orthogonality.

  • The most promising Ritz vectors

    If Vk = [v1, . . . , vk] spans the search subspace Vk,

    then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),

    i.e., uj ∈ Vk and ϑj ∈ C such that

    Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)

    or, equivalently, with Hk ≡ V∗kAVk,

    uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).

    Note the change in notation:we now suppress the iteration index k, while before we suppressed theordering index j in the set of all Ritz pairs at iteration step k:

    including the iteration index k as well in the present notation, the Ritz

    pairs (ϑk,uk) that we considered before would correspond to (ϑ(k)j ,u

    (k)j )

    for some j.

  • The most promising Ritz vectors

    If Vk = [v1, . . . , vk] spans the search subspace Vk,

    then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk),

    i.e., uj ∈ Vk and ϑj ∈ C such that

    Auj − ϑj uj ⊥ Vk (j = 1, . . . , k)

    or, equivalently, with Hk ≡ V∗kAVk,

    uj = Vk yj such that Hk yj = ϑj yj (j = 1, . . . , k).

    The ℓ most promising Ritz vectors u1, . . . ,uℓ are the ones

    with Ritz value that ‘best’ have the property that we want

    our wanted eigenvalue of A to have.

    Examples. • Re(ϑj) ≥ Re(ϑj+1) (j = 1, . . . , k − 1)

    if the eigenvalue of A with largest real part is wanted.

    • |ϑj − τ | ≤ |ϑj+1 − τ | (j = 1, . . . , k − 1)

    if the eigenvalue of A closest to some target τ ∈ C is wanted.

  • The most promising Ritz vectors

    If Vk = [v1, . . . , vk] spans the search subspace Vk,

    then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).

    For stability, we rather compute the Schur decomposition

    Hk = USU∗

    of Hk rather than the eigenvector decomposition:

    here U is k × k unitary and S is k × k upper triangular.

    The Ritz values are the diagonal entries of S: ϑj = Sjj.

    If Szj = ϑj zj then yj = U zj.

    Theorem. The first ℓ columns of U form an orthonormal

    basis of the space spanned by the eigenvectors y1, . . . , yℓ of

    Hk with eigenvalues the top ℓ diagonal entries S11, . . . , Sℓℓof S.

  • The most promising Ritz vectors

    If Vk = [v1, . . . , vk] spans the search subspace Vk,

    then we can compute k Ritz pairs (ϑ1,u1), . . . , (ϑk,uk).

    For stability, we rather compute the Schur decomposition

    Hk = USU∗

    of Hk rather than the eigenvector decomposition:

    here U is k × k unitary and S is k × k upper triangular.

    A Schur decomposition can be re-ordered (using unitary

    transforms, Givens rotations) such that the most promising

    Ritz values are the top diagonal entries of ‘new’ S:

    Theorem. There is a Schur decomposition Hk = USU∗ of

    Hk such that the diagonal elements of S appear in prescri-

    bed order.

    Assumption. u1, . . . ,uℓ are the most promising Ritz vec-

    tors and span(u1, . . . ,uℓ) = span(VkUe1, . . . ,VkUeℓ).

  • Restart

    If dim(V) = k is too high, k = kmax, then we have to

    restart to limit

    • high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),

    • high computational costs (to orthonormalise t against Vk).

    Simple restart. Take Ṽ = span(u1), i.e., Ṽ1 = [u1].

    Thick restart. Take Ṽ = span(u1, . . . ,uℓ) with ℓ = kmin

    (i.e., Ṽℓ = Vk U(:,1 : ℓ) ).

    Why thick restart? i.e., why ℓ = kmin > 1?

    • to (partially) maintain super linear convergence

    • to maintain a space that provides a good initial guess whenthe search to a next eigenpair is started.

    Explanation. The second Ritz pair is likely to converge to the secondeigenpair (recall the convergence proof of the power method). Themain component of the error in u1 is probably in the direction of u2,etc.

  • Restart

    If dim(V) = k is too high, k = kmax, then we have to

    restart to limit

    • high memory demands (to store Vk+1 = [v1, . . . ,vk,vk+1]),

    • high computational costs (to orthonormalise t against Vk).

    Simple restart. Take Ṽ = span(u1), i.e., Ṽ1 = [u1].

    Thick restart. Take Ṽ = span(u1, . . . ,uℓ) with ℓ = kmin

    (i.e., Ṽℓ = Vk U(:,1 : ℓ) ).

    Example. For the case where A∗ = A.

    Locally Optimal CG takes kmax = 3 and kmin = 2 and

    restarts with the two dimensional space spanned by the

    best Ritz vector and the best Ritz vector from the prece-

    ding step.LOCG expands with a Davidson step, solve t from Mt = r.

  • Restarting Arnoldi

    Arnoldi’s method requires a start with an Arnoldi relation.

    If, with η ≡ hk+1,k,

    AVk = Vk+1Hk = VkHk + ηvk+1e∗k,

    then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,

    AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)

    The ℓ-vector q will not be a multiple of eℓ and (∗) is not

    an Arnoldi relation.

  • Restarting Arnoldi

    Arnoldi’s method requires a start with an Arnoldi relation.

    If, with η ≡ hk+1,k,

    AVk = Vk+1Hk = VkHk + ηvk+1e∗k,

    then, with q∗ ≡ e∗kU(:,1 : ℓ) using Matlab’s notation,

    AVkU(:,1 : ℓ) = VkU(:,1 : ℓ)S + ηvk+1q∗. (∗)

    The ℓ-vector q will not be a multiple of eℓ and (∗) is not

    an Arnoldi relation.

    Theorem. There is a unitary matrix Q (product of Hou-

    seholder reflections) such that[Q 00∗ 1

    ]∗ [S

    ηq∗

    ]Q is (ℓ+1)× ℓ upper Hessenberg.

    Note. Except for vector updates Vk(U(:,1 : ℓ)Q) this allowsa restart at the cost of only low dimensional operations.

  • Implicitly Restarted Arnoldi Method

    More elegant and more efficient and stable implementation

    to compute U(:,1 : ℓ)Q by Sorenson is based on idea QR-

    algorithm.

  • Implicitly Restarted Arnoldi Method

    Select shifts µk, µk−1 . . . , µℓ+1 in C. Let p be the polynomial

    p(ζ) ≡ (ζ − µk) · . . . · (ζ − µℓ+1) (ζ ∈ C).

    Let ṽ1 ≡ p(A)v1. Consider the Arnoldi relation

    AṼℓ = Ṽℓ+1 H̃ℓ with Ṽℓe1 = ṽ1.

    Theorem. This Arnoldi relation can be obtained by ap-

    plying k − ℓ steps of the shifted QR-algorithm to Hk with

    shifts µj finding HkU = UH̃k and then use the first ℓ co-

    lumns of U to form Ṽℓ = VkU(:,1 : ℓ).

    IRAM selects µj = ϑj (the less best Ritz values). This

    gives the desired Arnoldi: the columns of Ṽℓ form an or-

    thonormal basis of the space spanned by u1, . . . ,uℓ. The

    eigenvalues of H̃ℓ are precisely ϑ1, . . . , ϑℓ.

  • Implicitly Restarted Arnoldi Method

    eigs.m in Matlab is IRAM (with deflation)

    eigs.m is based on ARPACK, a collection of FORTRAN

    routines that implements IRAM

    Lehoucq, Sorensen and Yang 1998

  • Program Lecture 4

    • Expansion

    ◦ Krylov subspace approach

    Lanczos, Arnoldi, Shift and Invert Arnoldi

    ◦ Convergence

    ◦ Accelerated Rayleigh Quotient Iteration

    Rational Krylov Sequence method

    ◦ Optimal expansion

    Jacobi-Davidson

    • Restart

    LOCG, Implicitly Restarted Arnoldi Method

    • Deflation

  • Towards the next eigenpair

    If an eigenvector x with eigenvalue λ has been detected

    (computed), how to continue the computational process

    for the next eigenpair?

  • Towards the next eigenpair

    If an eigenvector x with eigenvalue λ has been detected

    (computed), how to continue the computational process

    for the next eigenpair?

    1) Continue that process, but, at restart, do not select λ.

  • Towards the next eigenpair

    If an eigenvector x with eigenvalue λ has been detected

    (computed), how to continue the computational process

    for the next eigenpair?

    1) Continue that process, but, at restart, do not select λ.

    Very unstable: the computed eigenpair will be computed

    again, which is a waste of computational effort.

  • Towards the next eigenpair

    If an eigenvector x with eigenvalue λ has been detected

    (computed), how to continue the computational process

    for the next eigenpair?

    1) Continue that process, but, at restart, do not select λ.

    2) Deflation: remove the detected eigenvector x from the

    eigenvalue problem (and, in each step, from the computa-

    tional process).

  • Deflation

    For stability reason, aim for Schur decomposition

    AQ = QS

    with Q unitary and S upper triangular.

    Suppose partial Schur decomposition is available, i.e.,

    AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.

    Note. S11y = λy, then Ax = λx if x ≡Q1y.

  • Deflation

    For stability reason, aim for Schur decomposition

    AQ = QS

    with Q unitary and S upper triangular.

    Suppose partial Schur decomposition is available, i.e.,

    AQ1 = Q1S11 with Q1 the n× ℓ left block Q andS11 the ℓ× ℓ left upper block of S.

    Note. S11y = λy, then Ax = λx if x ≡Q1y.

    Theorem. q = qℓ+1 with λ = sℓ+1,ℓ+1 satisfies

    (I−Q1Q∗1)A(I−Q1Q

    ∗1)q = λq

  • Deflation

    Deflation can be incorporated in two ways.

    • Either by keeping the detected Schur vectors in the search

    subspace and working with A. This approach fits Arnoldi’s

    method well; see next transparency.

    • Or by removing these vectors from the search subspace

    and working with the deflated matrix. This approach fits

    Jacobi–Davidson well.

    In JD, it leads to the following JD correction equation:

    (I− [Q1,u][Q∗1,u])(A− ϑI)(I− [Q1,u][Q1,u]

    ∗)t = −r.

    Recall that enhancing the conditioning is one of the be-

    nefits of including a projection in the ‘Shift and Invert’

    equation. Including Q1 in the projections amplifies this ef-

    fect. Note that the solution t is orthogonal to Q1 and the

    search subspace gets automatically expanded with vectors

    ⊥Q1.

  • Deflating Arnoldi’s relation

    We want to retain an Arnoldi relation at deflation.

    Theorem. If AVk = Vk+1Hk, then the following properties

    are equivalent

    • One of the Ritz vectors uj is an eigenvector.

    • A sub-diagonal element, say hℓ+1,ℓ, of Hk is 0.

    • span(Vℓ) is A-invariant (whence, spanned by eigenvectors).

    Idea 1. Replace small sub-diagonal elements of Hk, say

    |hℓ+1,ℓ| ≤ ǫ, by 0.

    Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,

    use V(:, [ℓ+1 : k]) for selection (restart).

    Note. span(Vℓ) = span(Q1). If Vℓ is in the Arnoldi

    basis then orthonormalising Avk against Vk is equivalent

    to orthonormalising (I − Q1Q∗1)A(I − Q1Q

    ∗1)vk against

    V(:, [ℓ+1 : k]).

  • Deflating Arnoldi’s relation

    We want to retain an Arnoldi relation at deflation.

    Theorem. If AVk = Vk+1Hk, then the following properties

    are equivalent

    • One of the Ritz vectors uj is an eigenvector.

    • A sub-diagonal element, say hℓ+1,ℓ, of Hk is 0.

    • span(Vℓ) is A-invariant (whence, spanned by eigenvectors).

    Idea 1. Replace small sub-diagonal elements of Hk, say

    |hℓ+1,ℓ| ≤ ǫ, by 0.

    Idea 2. Keep Vℓ in the Arnoldi basis for orthogonalisation,

    use V(:, [ℓ+1 : k]) for selection (restart).

    Note. Idea 1 is unstable and needs stabilisation.

  • Example: JD + options

    The next transparencies illustrate Jacobi-Davidson with

    thick restart and deflation

  • 1. Start: Choose an initial z.

    • Set Λ = [ ], U = [ ], V = [ ].

    2. Repeat:

    a. V ← ModGS([V |z ]), W ← AV

    b. Compute H ← V∗W.

    c. Compute H = QRQ∗

    with Q∗Q = I, R upper ∆,

    |R11 − τ | ≤ |R22 − τ | ≤ . . ..

    d. ϑ ← R11, y ← Qe1, u ← Vy ,

    r ← ModGSU(Au− ϑu).

    e. If ‖ r ‖ ≤ tol

    • Λ ← [ Λ |ϑ ], U ← [U |u ].

    • J=[ e2 | . . . | edim(V) ],

    V← V(QJ), Q ← I, R← J∗RJ.

    • Goto 2.d.

    f . Solve (app.) z ⊥ Ũ ≡ [U |u ]

    (I− Ũ Ũ∗) (A− ϑ I) (I− Ũ Ũ

    ∗) z = −r.

  • -10

    -5

    0 10 20 30 40 50 60

    log

    10

    of

    resid

    ua

    l n

    orm

    number of flops x 1e6

    GMRES Ritz1

    -10

    -5

    log

    10

    of

    resid

    ua

    l n

    orm

    GMRES Harmonic1

    max(dim(span(V))) = 15, after restart dim(V) = 10

  • Spectrum MHD test problem

    −600

    −450

    −300

    −150

    0

    150

    300

    450

    600

    −50 −40 −30 −20 −10 0 10

    C

    Entire spectrum of the MHD test problem (n = 416)

    showing the correct location ( ) of the eigenvalues.

  • Alfvén spectrum

    0.0

    0.2

    0.4

    0.6

    0.8

    1.0

    −1.0 −0.8 −0.6 −0.4 −0.2 0.0

    C

    ∗�

    The MHD test problem (n = 416).

    Correct eigenvalues ( ◦ ), target ( ∗� ).