Eigenvalues in a nutshellEigenvalues in a nutshell

Mariquita Flores Garrido

UDLS, March 16th 2007

• Scalar multiple of a vector

• Addition of vectors

Just in case…

x

λx

xx x

λx

λx λx10 ≤≤ λ λ≤1 01 ≤≤− λ 1−≤λ

v1

v2

v1 + v2

Linear Transformations

mnnm RRfRA a:⇒∈ ×

V. gr.

• Rectangular matrices

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

635241

⎟⎟⎠

⎞⎜⎜⎝

⎛11

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

975

=

Ax = b Transformation of x by A.

n x 1

=A x Ax

m x n m x 1

*Stretch/Compression *Rotation

Linear Transformations

• Square Matrices

*Reflection

⎟⎟⎠

⎞⎜⎜⎝

⎛2002

⎟⎟⎠

⎞⎜⎜⎝

⎛0110

⎟⎟⎠

⎞⎜⎜⎝

⎛− ϕϕ

ϕϕcossinsincos

nnnn RRfRA a:⇒∈ × (*endomorphism)

*Shear in y-direction *Shear in x-direction

Bonnus: Shear

⎟⎟⎠

⎞⎜⎜⎝

⎛10

1 k⎟⎟⎠

⎞⎜⎜⎝

⎛101

k

x x

V.gr. Shear in x-direction

y y⎟⎟⎠

⎞⎜⎜⎝

⎛yx

⎟⎟⎠

⎞⎜⎜⎝

⎛ +ykyx

Basis for a Subspace

A basis in Rn is a set of n linearly independent vectors.

e3

e2

e1

2e3

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

211

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

211

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

001

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

010

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

100

= 1 + 1 + 2

Basis for a Subspace

Any set of n linearly independent vectors can be a basis

e1

e2 V1

V2

⎟⎟⎠

⎞⎜⎜⎝

⎛

2

1

aa

⎟⎟⎠

⎞⎜⎜⎝

⎛−=⎟⎟

⎠

⎞⎜⎜⎝

⎛12

2

1

aa

Using canonical basis:

??2

1 =⎟⎟⎠

⎞⎜⎜⎝

⎛aa

V1

V2

Using V1, V2 … ?

EIGENVALUES

•"Eigen" - "own", "peculiar to", "characteristic" or "individual“; "propervalue“.

• An invariant subspace under an endomorphism.

• If A is n x n matrix, x ≠ 0 is called an eigenvector of A if

Ax = λx

and λ is called an eigenvalue of A.

*Stretch/Compression *Rotation

Quiz 1

• Square Matrices (endomorphism)

*Reflection

⎟⎟⎠

⎞⎜⎜⎝

⎛2002

⎟⎟⎠

⎞⎜⎜⎝

⎛0110

⎟⎟⎠

⎞⎜⎜⎝

⎛− ϕϕ

ϕϕcossinsincos

• Characteristic polynomial: A degree n polynomial in λ:

det(λI - A) = 0Scalars satisfying the eqn, are the eigenvalues of A.

V.gr.

• Spectrum (of A) : { λ1, λ2 , …, λn}

• Algebraic multiplicity (of λi): number of roots equal to λi.

• Eigenspace (of λi): Eigenvectors never come alone!

• Geometric multiplicity (of λi): number of lin. independent eigenvectors associated with λi.

Eigen – slang

02543

214321 2 =−−=

−−

⎯→⎯⎟⎟⎠

⎞⎜⎜⎝

⎛λλ

λλ

)()( kxkxAxkAxk

xAx

λλ

λ

=⋅=⋅

=

Eigen – slang

• Eigen – something: Something that doesn’t change under some transformation.

xx

edxed

=][

FAQ (yeah, sure)

• How old are the eigenvalues?They arose before matrix theory, in the context of differential equations.

Bernoulli, Euler, 18th Century.

Hilbert, 20th century.

• Do all matrices have eigenvalues?Yes. Every n x n matrix has n eigenvalues.

• Why are the eigenvalues important?

- Physical meaning (v.gr. string, molecular orbitals ).

- There are other concepts relying on eigenvalues (v.gr. singular values, condition number).

- They tell almost everything about a matrix.

1. A singular ↔ λ = 0.

2. A and AT have the same λ’s.

3. A symmetric Real λ’s..

4. A skew-symmetric Imaginary λ’s..

5. A symmetric positive definite λ’s > 0

6. A full rank Eigenvectors form a basis for Rn.

7. A symmetric Eigenvectors can be chosen orthonormal.

8. A real Eigenvalues and eigenvectors come in conjugate pairs.

9. A symmetric Number of positive eigenvalues equals the number of positive pivots. A diagonal λi = aii

Properties of a matrix reflected in its eigenvalues:

10. A and M-1AM have the same λ’s.

11. A orthogonal all |λ | = 1

12. A projector λ = 1,0

13. A Markov λmax = 1

14. A reflection λ = -1,1,…,1

15. A rank one λ = vTu

16. A-1 1/λ(A)

17. A + cI λ(A) + c

18. A diagonal λi = aii

19. Eigenvectors of AAT Basis for Col(A)

20. Eigenvectors of ATA Basis for Row(A)

Properties of a matrix reflected in its eigenvalues:

M

What’s the worst thing about eigenvalues?

Find them is painful; they are roots of the characteristic polynomial.

* How long does it take to calculate the determinant of a 25 x 25 matrix?

* How do we find roots of polynomials?

WARNING:

The following examples have been simplified to be presented in a short

talk about eigenvalues. Attendee discretion is advised.

Example 1: Face Identification

Eigenfaces: face identification technique.

(There are also eigeneyes, eigennoses, eigenmouths, eigenears,eigenvoices,…)

EIGENFACES

Given a set of images, and a “target face”, identify the

“owner” of the face.

128 images

(train set)

256 x 256

(test)

1. Preprocessing stage: linear transformations, morphing, warping,…

2. Representing faces: vectors (Γj) in a very high dimensional space.

V.gr.

Training set: 65536 x 128 matrix

3. Centering data: take the “average” image and define every Φj

∑=

Γ=Ψn

jjn 1

1

jj Γ−Ψ=Φ

],...,,[ 21 nA ΦΦΦ=

4. Eigenvectors of AAT are a basis for Col(A) (what’s the size of this matrix?), so instead of working with A, I can express every image in another basis.

* 5. PCA: reducing the dimension of the space. To solve the problem, the work is done in a smaller subspace, SL, using projections of each image onto SL.

6. It’s possible to get eigenvectors of AAT using eigenvectors of ATA.65436 x 65436 128 x 128

Example 2: Sparse Matrix Computations

ITERATIVE METHODS

 x = b

• Gauss-Jordan

• If  is 105 ×105 , Gauss Jordan would take approx. 290 years.

• Iterative methods: find some “good” matrix A and apply it to some initial vector until you get convergence.

• Choosing different A determines different methods (v.gr. Jacobi, Gauss-Seidel, Krylov subspace methods, …).

Example 2: ITERATIVE METHODS

0n

n

02

012

01

xA x

xA )A(Ax Ax x

Ax x

=

===A: huge matrix ( 106 ×106 )

x0 : initial guess

mn

mnn

mn

mnn

mmnn

vvv

vAvAvA

vvvAxA

mλαλαλα

ααα

ααα

+++=

+++=

+++=

L

L

L

22211

2211

22110

1

)(

=

M

econvergenci ⇒<1λ

• If A has full rank, its eigenvectors form a basis for Rm

• Iteration

Convergence, number of iterations, it’s all about eigenvalues…

Example 2: ITERATIVE METHODS

Example 3: Dynamical Systems

( Eigenvalues don’t have the main role here, but, who are you going to complain to?)

Arnold’s Cat

• Poincare recurrence theorem:

“ A system having a finite amount of energy and confined to a finite spatial volume will, after a sufficiently long time, return to an arbitrarily small neighborhood of its initial state.”

• Vladimir I. Arnold, Russian mathematician.

⎟⎟⎠

⎞⎜⎜⎝

⎛=

2111

A

Each pixel can be assigned to a unique pair of coordinates

(a two-dimensional vector)

⎟⎟⎠

⎞⎜⎜⎝

⎛⋅⎟⎟

⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛=

1011

1101

2111

A(mod 1)

1 2 3 5

20 31 37 42

46 47 59 63

80797877

⎟⎟⎠

⎞⎜⎜⎝

⎛=

2111

A⎟⎟⎠

⎞⎜⎜⎝

⎛→=

85.52.

61.21λ

⎟⎟⎠

⎞⎜⎜⎝

⎛−→=

52.85.

38.02λ

1)det( =A V1

V2

More Applications

•Graph theory

•Differential Equations

•PageRank

•Physics

REFERENCES

•Chen Greif. CPSC 517 Notes, UBC/CS, Spring 2007.

•Howard Anton and Chris Rorres. Elementary Linear Algebra, Applications Version, 9th Ed. John Wiley & Sons, Inc. 2005

•Humberto Madrid de la Vega. Eigenfaces: Reconocimiento digital de facciones mediante SVD. Memorias del XXXVII Congreso SMM, 2005.

•Wikipedia: Eigenvalue, eigenvector and eigenspace.http://en.wikipedia.org/wiki/Eigenvalue