Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR...

38
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Eigenproblems II: Computation 1 / 27

Transcript of Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR...

Page 1: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Eigenproblems II: Computation

CS 205A:Mathematical Methods for Robotics, Vision, and Graphics

Justin Solomon

CS 205A: Mathematical Methods Eigenproblems II: Computation 1 / 27

Page 2: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Setup

A ∈ Rn×n symmetric

~x1, . . . , ~xn ∈ Rn eigenvectors

|λ1| ≥ |λ2| ≥ · · · ≥ |λn| eigenvalues

Review (Spectral Theorem):What do we know about the eigenvectors?

CS 205A: Mathematical Methods Eigenproblems II: Computation 2 / 27

Page 3: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Setup

A ∈ Rn×n symmetric

~x1, . . . , ~xn ∈ Rn eigenvectors

|λ1| ≥ |λ2| ≥ · · · ≥ |λn| eigenvalues

Review (Spectral Theorem):What do we know about the eigenvectors?

CS 205A: Mathematical Methods Eigenproblems II: Computation 2 / 27

Page 4: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Usual Trick

~v ∈ Rn

⇓~v = c1~x1 + · · · + cn~xn

CS 205A: Mathematical Methods Eigenproblems II: Computation 3 / 27

Page 5: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Observation

Ak~v =

λk1

(c1~x1 +

(λ2λ1

)kc1~x2 + · · · +

(λnλ1

)kcn~xn

)

CS 205A: Mathematical Methods Eigenproblems II: Computation 4 / 27

Page 6: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

For Large k

Ak~v ≈ λk1~x1

(assuming |λ2| < |λ1|)

CS 205A: Mathematical Methods Eigenproblems II: Computation 5 / 27

Page 7: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Power Iteration

~vk = A~vk−1

Question:What if |λ1| > 1?

CS 205A: Mathematical Methods Eigenproblems II: Computation 6 / 27

Page 8: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Power Iteration

~vk = A~vk−1

Question:What if |λ1| > 1?

CS 205A: Mathematical Methods Eigenproblems II: Computation 6 / 27

Page 9: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Normalized Power Iteration

~wk = A~vk−1

~vk =~wk‖~wk‖

Question: Which norm?

CS 205A: Mathematical Methods Eigenproblems II: Computation 7 / 27

Page 10: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Normalized Power Iteration

~wk = A~vk−1

~vk =~wk‖~wk‖

Question: Which norm?

CS 205A: Mathematical Methods Eigenproblems II: Computation 7 / 27

Page 11: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Eigenvalues of Inverse Matrix

A~v = λ~v =⇒ A−1~v =1

λ~v

Question:What is the largest-magnitude eigenvalue?

CS 205A: Mathematical Methods Eigenproblems II: Computation 8 / 27

Page 12: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Eigenvalues of Inverse Matrix

A~v = λ~v =⇒ A−1~v =1

λ~v

Question:What is the largest-magnitude eigenvalue?

CS 205A: Mathematical Methods Eigenproblems II: Computation 8 / 27

Page 13: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Inverse Iteration

~wk = A−1~vk−1

~vk =~wk‖~wk‖

Question: How to make faster?

CS 205A: Mathematical Methods Eigenproblems II: Computation 9 / 27

Page 14: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Inverse Iteration

~wk = A−1~vk−1

~vk =~wk‖~wk‖

Question: How to make faster?

CS 205A: Mathematical Methods Eigenproblems II: Computation 9 / 27

Page 15: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Inverse Iteration with LU

Solve L~yk = ~vk−1

Solve U ~wk = ~yk

Normalize ~vk =~wk‖~wk‖

CS 205A: Mathematical Methods Eigenproblems II: Computation 10 / 27

Page 16: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Eigenvalues of Inverse Matrix

A~v = λ~v =⇒ (A− σI)~v = (λ− σ)~v

CS 205A: Mathematical Methods Eigenproblems II: Computation 11 / 27

Page 17: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Shifted Inverse Iteration

To find eigenvalue closest to σ:

~vk+1 =(A− σI)−1~vk‖(A− σI)−1~vk‖

CS 205A: Mathematical Methods Eigenproblems II: Computation 12 / 27

Page 18: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Heuristic: Convergence Rate

Ak~v =

λk1

(c1~x1 +

(λ2λ1

)kc1~x2 + · · · +

(λnλ1

)kcn~xn

)

CS 205A: Mathematical Methods Eigenproblems II: Computation 13 / 27

Page 19: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Strategy for Better Convergence

Find σ with∣∣∣∣λ2 − σλ1 − σ

∣∣∣∣ < ∣∣∣∣λ2λ1∣∣∣∣

CS 205A: Mathematical Methods Eigenproblems II: Computation 14 / 27

Page 20: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Least-Squares Approximation

If ~v0 is approximately an eigenvector:

argminλ‖A~v0 − λ~v0‖2 =

~v>0 A~v0‖~v0‖22

CS 205A: Mathematical Methods Eigenproblems II: Computation 15 / 27

Page 21: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Rayleigh Quotient Iteration

σk =~v>k−1A~vk−1

‖~vk−1‖22~wk = (A− σkI)−1~vk−1

~vk =~wk‖~wk‖

Efficiency per iteration vs. number of iterations?

CS 205A: Mathematical Methods Eigenproblems II: Computation 16 / 27

Page 22: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Rayleigh Quotient Iteration

σk =~v>k−1A~vk−1

‖~vk−1‖22~wk = (A− σkI)−1~vk−1

~vk =~wk‖~wk‖

Efficiency per iteration vs. number of iterations?

CS 205A: Mathematical Methods Eigenproblems II: Computation 16 / 27

Page 23: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Unlikely Failure Mode for Iteration

What is ~v0?

What happens when~v0 · ~x1 = 0?

CS 205A: Mathematical Methods Eigenproblems II: Computation 17 / 27

Page 24: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Unlikely Failure Mode for Iteration

What is ~v0?

What happens when~v0 · ~x1 = 0?

CS 205A: Mathematical Methods Eigenproblems II: Computation 17 / 27

Page 25: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Bug or Feature?

1. Compute ~x0 via power iteration.

2. Project ~x0 out of ~v0.

3. Compute ~x1 via power iteration.

4. Project span {~x0, ~x1} out of ~v0.

5. · · ·

Assumption: A is symmetric.

CS 205A: Mathematical Methods Eigenproblems II: Computation 18 / 27

Page 26: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Bug or Feature?

1. Compute ~x0 via power iteration.

2. Project ~x0 out of ~v0.

3. Compute ~x1 via power iteration.

4. Project span {~x0, ~x1} out of ~v0.

5. · · ·

Assumption: A is symmetric.

CS 205A: Mathematical Methods Eigenproblems II: Computation 18 / 27

Page 27: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Avoiding Numerical Drift

Do power iteration on P>AP where P projects

out known eigenvectors.

DeflationModify A so that power iteration reveals an

eigenvector you have not yet computed.

CS 205A: Mathematical Methods Eigenproblems II: Computation 19 / 27

Page 28: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Similarity Transformations

Similar matricesTwo matrices A and B are similar if there exists

T with B = T−1AT.

PropositionSimilar matrices have the same eigenvalues.

CS 205A: Mathematical Methods Eigenproblems II: Computation 20 / 27

Page 29: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Similarity Transformations

Similar matricesTwo matrices A and B are similar if there exists

T with B = T−1AT.

PropositionSimilar matrices have the same eigenvalues.

CS 205A: Mathematical Methods Eigenproblems II: Computation 20 / 27

Page 30: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Householder Asymmetric Deflation

H~x1 = ~e1

=⇒ HAH>~e1 = HAH~e1 by symmetry

= HA~x1 since H2 = I

= λ1H~x1

= λ1~e1

CS 205A: Mathematical Methods Eigenproblems II: Computation 21 / 27

Page 31: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Householder Asymmetric Deflation

HAH> =

(λ1 ~b

>

~0 B

)Similarity transform of A =⇒ same eigenvalues.

Do power iteration on B.

CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 27

Page 32: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Householder Asymmetric Deflation

HAH> =

(λ1 ~b

>

~0 B

)Similarity transform of A =⇒ same eigenvalues.

Do power iteration on B.

CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 27

Page 33: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

<warning>

Justin’s favorite

algorithm. Ever.

</warning>

CS 205A: Mathematical Methods Eigenproblems II: Computation 23 / 27

Page 34: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Conjugation without Inversion

Q−1 = Q>

=⇒ Q−1AQ = Q>AQ

But which Q?Should involve matrix structure but be easy to

compute.

CS 205A: Mathematical Methods Eigenproblems II: Computation 24 / 27

Page 35: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Conjugation without Inversion

Q−1 = Q>

=⇒ Q−1AQ = Q>AQ

But which Q?Should involve matrix structure but be easy to

compute.

CS 205A: Mathematical Methods Eigenproblems II: Computation 24 / 27

Page 36: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

Experiment

A = QR

Q−1AQ = ?

CS 205A: Mathematical Methods Eigenproblems II: Computation 25 / 27

Page 37: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

QR Iteration

A1 = A

Factor Ak = QkRk

Multiply Ak+1 = RkQk

CS 205A: Mathematical Methods Eigenproblems II: Computation 26 / 27

Page 38: Eigenproblems II: Computation · Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Eigenproblems II: Computation CS 205A: Mathematical Methods for Robotics, Vision,

Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration

If QR Iteration Converges

A∞ = Q∞R∞ = R∞Q∞

I Eigenvalues of R∞ along the diagonal and

are eigenvalues of A∞ (R∞Q∞)

I Eigenvalues of A∞ are those of A

Next

CS 205A: Mathematical Methods Eigenproblems II: Computation 27 / 27