Numerical Analysis Massoud Malek

Vector and Matrix Norms

♠ Vector Norm. Given an n-dimensional vector x =

x1

x2...

xn

, a vector norm ||x||, is a non-

negative real number, defined such that

1. ‖x‖ > 0 when x 6= 0 and ||x|| = 0 if and only if x = θ, the zero vector,

2. ||kx|| = |k| × ||x|| for any scalar k,

3. ||x + y|| ≤ ||x||+ ||y||,

For p = 1, 2, . . . ,∞, the p-vector norm ||x‖p is defined as

‖x‖p =

[∑

i

|xi|p]1/p

.

The special case ‖x‖∞ is defined as

‖x‖∞ = maxi|xi| .

The most commonly encountered vector norm (often simply called ”the norm” of a vector)

is the L2-norm, given by

‖x‖2 = ‖x‖ =√

x21 + x2

2 + ... + x2n.

This and other types of vector norms are summarized in the following table, together

with the value of the norm for the example vector v = (1, 2, 3)t.

norm symbol value approx.

L1 ||x||1 6 6.000L2 ||x||2

√14 3.742

L3 ||x||3 62/3 3.302L4 ||x||4 21/4

√7 3.146

L∞ ||x||∞ 3 3.000

The concept of unit circle (the set of all vectors of norm 1) is different in different norms:

for the 1-norm the unit circle in IR2 is a square with vertices at (1, 0), (0, 1), (−1, 0), and

(0,−1); for the 2-norm (Euclidean norm) it is the well-known unit circle, while for the

infinity norm it is a square with vertices at (1, 1), (−1, 1), (−1,−1), and (1,−1). Note that

due to the definition of the norm, the unit circle is always convex and centrally symmetric

(therefore, for example, the unit ball may be a rectangle but cannot be a triangle).

Two norms || · ||α and || · ||β on a vector space V are called equivalent if there exist positive

real numbers R and S such that

R‖x‖α ≤ ‖x‖β ≤ S‖x‖α,

California State University, East Bay

Numerical Analysis Vector and Matrix Norms Page 2

for all x ∈ V . On a finite-dimensional vector space all norms are equivalent. For instance,

the L1, L2, and L∞ norms are all equivalent on IRn:

‖x‖2 ≤ ‖x‖1 ≤√

n ‖x‖2‖x‖∞ ≤ ‖x‖2 ≤

√n ‖x‖∞

‖x‖∞ ≤ ‖x‖1 ≤ n ‖x‖∞ .

Equivalent norms define the same notions of continuity and convergence and for many

purposes do not need to be distinguished. To be more precise the uniform structure

defined by equivalent norms on the vector space is uniformly isomorphic.

Given any norm || · ||α and any non-singular matrix A, we can define a vector norm || · ||βsuch that ||x||β = ||Ax||α .

♠ Matrix Norm. Given a complex or real matrix A = (aij), a matrix norm ||A|| is a non-

negative number associated with A having the properties:

1. ||A|| > 0 when A 6= 0 and ||A|| = 0 if and only if A = Z, the zero matrix,

2. ||kA|| = |k| × ||A|| for any scalar k,

3. ||A + B|| ≤ ||A||+ ||B||,

For square matrices, a sub-multiplicative matrix norm also satisfies:

4. ||AB|| ≤ ||A|| × ||B||.

If a matrix norm is not sub-multiplicative, then it is called a generalized matrix norm.

Let λ1, λ2, ..., λn be the eigenvalues of A, then

1||A−1|| ≤ |λ| ≤ ||A||. (1)

The spectrum of a square matrix A, denoted by σ(A) is the set of all eigenvalues of A.

The spectral radius of A, denoted by ρ(A) is defined as:

ρ(A) = max{|λ| : λ ∈ σ(A)}

The spectral norm of m × n matrix A, denoted by ||A||2, which is the square root of the

maximum eigenvalue of the positive semi-definite matrix A∗A or AA∗ (we choose the one

in smaller size),

||A||2 =√

ρ(A∗A) =√

ρ(AA∗) (2)

is often referred to as ”the” matrix norm.

The row norm is defined by

||A||∞ = maxi

n∑

j=1

|aij |. (3)

California State University, East Bay

Numerical Analysis Vector and Matrix Norms Page 3

||A||1, ||A||2, and ||A||∞ satisfy the inequality

||A||22 ≤ ||A||1 × ||A||∞.

The Frobenius norm of an m × n matrix A defined as the square root of the sum of the

absolute squares of its elements; it is also equal to the square root of the trace of the

positive semi-definite matrix AA∗

||A||F =

√√√√m∑

i=1

n∑

j=1

|aij |2 =√

trace(AA∗) . (4)

The Frobenius norm can also be considered as a vector norm.

♣ Induced Norm. If vector norms on IKn are given (IKn is field of real or complex numbers),

then one defines the corresponding induced norm, natural norm, or operator norm on

the space of matrices as the follows:

‖A‖ = max{‖Ax‖ : x ∈ IKn with ‖x‖ ≤ 1}= max{‖Ax‖ : x ∈ IKn with ‖x‖ = 1}

= max{‖Ax‖‖x‖ : x ∈ IKn with x 6= θ

}.

For example, the operator norm corresponding to the p-norm for vectors is:

‖A‖p = maxx 6=0

‖Ax‖p

‖x‖p

.

In the case of p = 1 and p = ∞, the norms can be computed as:

‖A‖1 = max1≤j≤n

m∑

i=1

|aij |,

as the column norm of the matrix A and

‖A‖∞ = max1≤i≤m

n∑

j=1

|aij |,

as the row norm of the matrix A. In the case of p = 2 (the Euclidean norm), the induced

matrix norm is the spectral norm.

Theorem 1. All induced norms are sub-multiplicative.

Proof. Let A and B be two matrices where AB is well defined, then

||AB|| = maxx 6=θ

‖ABx‖‖x‖ = max

x 6=θ

‖ABx‖‖Bx‖

‖Bx‖‖x‖ ≤

[maxx 6=θ

‖Ax‖‖x‖

] [maxx 6=θ

‖bx‖‖x‖

]= ‖A‖ × ‖B‖.

California State University, East Bay

Numerical Analysis Vector and Matrix Norms Page 4

Theorem 2. If A is an n× n, then ρ(A) ≤ ||A|| for any sub-multiplicative matrix norm || · ||.

Proof. Let v be a eigenvector of A corresponding to the dominant eigenvalue λ

(i.e., Av = λv with |λ| = ρ(A) ). Define a matrix V , where the first column of V is v

and the next n− 1 columns are zero vectors. Define the matrix

U =1‖V ‖V with the property AU = λU ;

then we have

ρ(A) = |λ| . 1 = |λ| ||U || = ||λU || = ||AU || ≤ ||A|| ‖U‖ = ‖A‖.

Note. The max norm ‖A‖max = max{|aij |} is not a sub-multiplicative matrix norm. Also

the sub-multiplicative Frobenius norm is not an induced norm since the induced norm of

any n× n identity matrix In is one, but || In||F =√

n.

A matrix norm ‖ · ‖ab on IKm×n is called consistent with a vector norm ‖ · ‖a on IKn and

a vector norm ‖ · ‖b on IKm if:

‖Ax‖b ≤ ‖A‖ab ‖x‖a

for all A ∈ IKm×n, x ∈ IKn. All induced norms are consistent by definition.

♣ Equivalence of Norms. For any two vector norms || · ||α and || · ||β, we have

R‖A‖α ≤ ‖A‖β ≤ S ‖A‖α

for some positive numbers R and S, for all matrices A ∈ IKm×n. Moreover, when A ∈ IRn×n,

then for any vector norm ‖ · ‖, there exists a unique positive number K such that L‖A‖is a sub-multiplicative matrix norm for every L ≥ K. A matrix norm || · ||a is said to be

minimal if there exists no other matrix norm || · ||b satisfying || · ||a ≤ || · ||b.

Examples of norm equivalence. For a matrix A ∈ IRm×n, the following inequalities hold:

‖A‖2 ≤ ‖A‖F ≤ √n ‖A‖2

‖A‖max ≤ ‖A‖2 ≤√

mn ‖A‖max

1√n‖A‖∞ ≤ ‖A‖2 ≤

√m ‖A‖∞

1√m‖A‖1 ≤ ‖A‖2 ≤

√n ‖A‖1 .

Example Let

A =

1 −2 3−4 5 −67 −8 9

, B =

1 −1 02 3 −15 0 −2

, and C =

1 1 22 1 11 2 1

.

California State University, East Bay

Numerical Analysis Vector and Matrix Norms Page 5

Then for A, we obtain:

ρ(A) ≈ |16.1168| = 16.1168,

‖A‖∞ = max{ |1|+ | − 2|+ |3|, | − 4|+ |5|+ | − 6|, |7|+ | − 8|+ |9| } = 24,

‖A‖1 = max{ |1|+ | − 4|+ |7|, | − 2|+ |5|+ | − 8|, |3|+ | − 6|+ |9| } = 18,

‖A‖2 =√

ρ(AA∗) ≈√

283.8585 ≈ 16.8481,

‖A‖F =√|1|2 + | − 2|2 + |3|2 + | − 4|2 + |5|2 + | − 6|2 + |7|2 + | − 8|2 + |9|2 =

√285 ≈ 16.8819,

‖A‖max = max{ |1|, | − 2|, |3|, | − 4|, |5|, | − 6|, |7|, | − 8|, |9|} = |9| ,

with

‖A‖max < ρ(A) < ‖A‖2 < ‖A‖F < ‖A‖1 < ‖A‖∞ .

Now for B, we obtain:

ρ(B) ≈ |2.3772| = 2.3772,

‖B‖∞ = max{ |1|+ | − 1|+ |0|, |2|+ |3|+ | − 1|, |5|+ |0|+ | − 2| } = 7,

‖B‖1 = max{ |1|+ |2|+ |5|, | − 1|+ |3|+ |0|, |0|+ | − 1|+ | − 2| } = 8,

‖B‖2 =√

ρ(BB∗)) ≈√

36.0994 ≈ 6.0083,

‖B‖F =√|1|2 + | − 1|2 + |0|2 + |2|2 + |3|2 + | − 1|2 + |5|2 + |0|2 + | − 2|2 =

√45 ≈ 6.7082,

‖B‖max = max{ |1|, | − 1|, |0|, |2|, |3|, | − 1|, |5|, |0|, | − 2| } = |5| ,

ρ(B) < ‖B‖max < ‖B‖2 < ‖B‖F < ‖B‖∞ < ‖B‖1 .

Finally, for C, we obtain:

ρ(C) = |4| = 4,

‖C‖∞ = max{ |1|+ |1|+ |2|, |2|+ |1|+ |1|, |1|+ |2|+ |1| } = 4,

‖C‖1 = max{ |1|+ |2|+ |1|, |1|+ |1|+ |2|, |2|+ |1|+ |1| } = 4,

‖C‖2 =√

ρ(CC∗)) =√

16 = 4,

‖C‖F =√|1|2 + |1|2 + |2|2 + |1|2 + |1|2 + |2|2 + |1|2 + |2|2 + |1|2 =

√18 ≈ 4.2426,

‖C‖max = max{ |1|, |1|, |2|, |2|, |1|, |1|, |1|, |2|, |1| } = |2| ,

|C‖max < ρ(C) = ‖C‖1 = ‖C‖2 = ‖C‖∞ < ‖C‖F .

California State University, East Bay