Lecture 1: Inner product, Orthogonality, Vector/Matrix...
17
Lecture 1: Inner product, Orthogonality, Vector/Matrix norm September 15, 2020 Numerical Linear Algebra Lecture 1 September 15, 2020 1 / 17
Transcript of Lecture 1: Inner product, Orthogonality, Vector/Matrix...
Lecture 1 Inner product OrthogonalityVectorMatrix norm
September 15 2020
Numerical Linear Algebra Lecture 1 September 15 2020 1 17
1 Inner product on a linear space V over a number field F (C or R)Definition A function 〈middot middot〉 Vtimes V rarr F is called an inner productif it satisfies the following three conditions (forall xy z isin V forall α isin F)(1) Conjugate symmetry
〈xy〉 = 〈yx〉
(2) Positive definiteness
〈xx〉 ge 0 〈xx〉 = 0 hArr x = 0
(3) Linearity in the first variable
〈x+ y z〉 = 〈x z〉+ 〈y z〉 〈αxy〉 = α〈xy〉
Example the standard inner product on the space V = Cm
forallxy isin Cm 〈xy〉 = ylowastx =983131m
i=1xiyi
Numerical Linear Algebra Lecture 1 September 15 2020 2 17
Example the A-inner product on the space V = Cm
forallxy isin Cm 〈xy〉 = ylowastAx
where A is a given hermitian positive definite matrix
2 Orthogonality
Orthogonality is a mathematical concept with respect to a giveninner product 〈middot middot〉(1) Two vectors x and y are called orthogonal if 〈xy〉 = 0(2) Two sets of vectors X and Y are called orthogonal if forallx isin X
and y isin Y 〈xy〉 = 0(3) A set of nonzero vectors S is orthogonal if forallxy isin S and
x ∕= y 〈xy〉 = 0 if further forallx isin S 〈xx〉 =1 S isorthonormal
Proposition 1
The vectors in an orthogonal set S are linearly independent
Numerical Linear Algebra Lecture 1 September 15 2020 3 17
21 Orthogonal components of a vector
Inner products can be used to decompose arbitrary vectors intoorthogonal components Given an orthonormal set q1q2 qnand an arbitrary vector v let
r = v minus 〈vq1〉q1 minus 〈vq2〉q2 minus middot middot middotminus 〈vqn〉qn
Obviouslyr isin spanq1q2 qnperp
Thus we see that v can be decomposed into n+ 1 orthogonalcomponents
v = r+ 〈vq1〉q1 + 〈vq2〉q2 + middot middot middot+ 〈vqn〉qn
We call 〈vqi〉qi the part of v in the direction of qi and r thepart of v orthogonal to the subspace spanq1q2 qn
Numerical Linear Algebra Lecture 1 September 15 2020 4 17
Exercise Write the expression for v when the set q1q2 qnis only orthogonal
Cauchy-Schwarz inequality For any given inner product 〈middot middot〉
|〈xy〉| le983155
〈xx〉983155
〈yy〉
The equality holds if and only if x and y are linearly dependent
Exercise Prove the inequality Hint write
x =〈xy〉〈yy〉y + z
Then 〈zy〉 = 0 Consider 〈xx〉Application For any hermitian positive definite matrix A
|ylowastAx|2 le (xlowastAx)(ylowastAy)
Numerical Linear Algebra Lecture 1 September 15 2020 5 17
3 Norm on a linear space V over a number field F (C or R)Definition A function 983042 middot 983042 V rarr R is called a norm if it satisfiesthe following three conditions (forall xy isin V and forall α isin F)(1) Nonnegativity
983042x983042 ge 0 983042x983042 = 0 hArr x = 0
(2) Positive homogeneity
983042αx983042 = |α|983042x983042
(3) Triangle inequality
983042x+ y983042 le 983042x983042+ 983042y983042
Exercise Show that a norm is a continuous function
Numerical Linear Algebra Lecture 1 September 15 2020 6 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
1 Inner product on a linear space V over a number field F (C or R)Definition A function 〈middot middot〉 Vtimes V rarr F is called an inner productif it satisfies the following three conditions (forall xy z isin V forall α isin F)(1) Conjugate symmetry
〈xy〉 = 〈yx〉
(2) Positive definiteness
〈xx〉 ge 0 〈xx〉 = 0 hArr x = 0
(3) Linearity in the first variable
〈x+ y z〉 = 〈x z〉+ 〈y z〉 〈αxy〉 = α〈xy〉
Example the standard inner product on the space V = Cm
forallxy isin Cm 〈xy〉 = ylowastx =983131m
i=1xiyi
Numerical Linear Algebra Lecture 1 September 15 2020 2 17
Example the A-inner product on the space V = Cm
forallxy isin Cm 〈xy〉 = ylowastAx
where A is a given hermitian positive definite matrix
2 Orthogonality
Orthogonality is a mathematical concept with respect to a giveninner product 〈middot middot〉(1) Two vectors x and y are called orthogonal if 〈xy〉 = 0(2) Two sets of vectors X and Y are called orthogonal if forallx isin X
and y isin Y 〈xy〉 = 0(3) A set of nonzero vectors S is orthogonal if forallxy isin S and
x ∕= y 〈xy〉 = 0 if further forallx isin S 〈xx〉 =1 S isorthonormal
Proposition 1
The vectors in an orthogonal set S are linearly independent
Numerical Linear Algebra Lecture 1 September 15 2020 3 17
21 Orthogonal components of a vector
Inner products can be used to decompose arbitrary vectors intoorthogonal components Given an orthonormal set q1q2 qnand an arbitrary vector v let
r = v minus 〈vq1〉q1 minus 〈vq2〉q2 minus middot middot middotminus 〈vqn〉qn
Obviouslyr isin spanq1q2 qnperp
Thus we see that v can be decomposed into n+ 1 orthogonalcomponents
v = r+ 〈vq1〉q1 + 〈vq2〉q2 + middot middot middot+ 〈vqn〉qn
We call 〈vqi〉qi the part of v in the direction of qi and r thepart of v orthogonal to the subspace spanq1q2 qn
Numerical Linear Algebra Lecture 1 September 15 2020 4 17
Exercise Write the expression for v when the set q1q2 qnis only orthogonal
Cauchy-Schwarz inequality For any given inner product 〈middot middot〉
|〈xy〉| le983155
〈xx〉983155
〈yy〉
The equality holds if and only if x and y are linearly dependent
Exercise Prove the inequality Hint write
x =〈xy〉〈yy〉y + z
Then 〈zy〉 = 0 Consider 〈xx〉Application For any hermitian positive definite matrix A
|ylowastAx|2 le (xlowastAx)(ylowastAy)
Numerical Linear Algebra Lecture 1 September 15 2020 5 17
3 Norm on a linear space V over a number field F (C or R)Definition A function 983042 middot 983042 V rarr R is called a norm if it satisfiesthe following three conditions (forall xy isin V and forall α isin F)(1) Nonnegativity
983042x983042 ge 0 983042x983042 = 0 hArr x = 0
(2) Positive homogeneity
983042αx983042 = |α|983042x983042
(3) Triangle inequality
983042x+ y983042 le 983042x983042+ 983042y983042
Exercise Show that a norm is a continuous function
Numerical Linear Algebra Lecture 1 September 15 2020 6 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Example the A-inner product on the space V = Cm
forallxy isin Cm 〈xy〉 = ylowastAx
where A is a given hermitian positive definite matrix
2 Orthogonality
Orthogonality is a mathematical concept with respect to a giveninner product 〈middot middot〉(1) Two vectors x and y are called orthogonal if 〈xy〉 = 0(2) Two sets of vectors X and Y are called orthogonal if forallx isin X
and y isin Y 〈xy〉 = 0(3) A set of nonzero vectors S is orthogonal if forallxy isin S and
x ∕= y 〈xy〉 = 0 if further forallx isin S 〈xx〉 =1 S isorthonormal
Proposition 1
The vectors in an orthogonal set S are linearly independent
Numerical Linear Algebra Lecture 1 September 15 2020 3 17
21 Orthogonal components of a vector
Inner products can be used to decompose arbitrary vectors intoorthogonal components Given an orthonormal set q1q2 qnand an arbitrary vector v let
r = v minus 〈vq1〉q1 minus 〈vq2〉q2 minus middot middot middotminus 〈vqn〉qn
Obviouslyr isin spanq1q2 qnperp
Thus we see that v can be decomposed into n+ 1 orthogonalcomponents
v = r+ 〈vq1〉q1 + 〈vq2〉q2 + middot middot middot+ 〈vqn〉qn
We call 〈vqi〉qi the part of v in the direction of qi and r thepart of v orthogonal to the subspace spanq1q2 qn
Numerical Linear Algebra Lecture 1 September 15 2020 4 17
Exercise Write the expression for v when the set q1q2 qnis only orthogonal
Cauchy-Schwarz inequality For any given inner product 〈middot middot〉
|〈xy〉| le983155
〈xx〉983155
〈yy〉
The equality holds if and only if x and y are linearly dependent
Exercise Prove the inequality Hint write
x =〈xy〉〈yy〉y + z
Then 〈zy〉 = 0 Consider 〈xx〉Application For any hermitian positive definite matrix A
|ylowastAx|2 le (xlowastAx)(ylowastAy)
Numerical Linear Algebra Lecture 1 September 15 2020 5 17
3 Norm on a linear space V over a number field F (C or R)Definition A function 983042 middot 983042 V rarr R is called a norm if it satisfiesthe following three conditions (forall xy isin V and forall α isin F)(1) Nonnegativity
983042x983042 ge 0 983042x983042 = 0 hArr x = 0
(2) Positive homogeneity
983042αx983042 = |α|983042x983042
(3) Triangle inequality
983042x+ y983042 le 983042x983042+ 983042y983042
Exercise Show that a norm is a continuous function
Numerical Linear Algebra Lecture 1 September 15 2020 6 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
21 Orthogonal components of a vector
Inner products can be used to decompose arbitrary vectors intoorthogonal components Given an orthonormal set q1q2 qnand an arbitrary vector v let
r = v minus 〈vq1〉q1 minus 〈vq2〉q2 minus middot middot middotminus 〈vqn〉qn
Obviouslyr isin spanq1q2 qnperp
Thus we see that v can be decomposed into n+ 1 orthogonalcomponents
v = r+ 〈vq1〉q1 + 〈vq2〉q2 + middot middot middot+ 〈vqn〉qn
We call 〈vqi〉qi the part of v in the direction of qi and r thepart of v orthogonal to the subspace spanq1q2 qn
Numerical Linear Algebra Lecture 1 September 15 2020 4 17
Exercise Write the expression for v when the set q1q2 qnis only orthogonal
Cauchy-Schwarz inequality For any given inner product 〈middot middot〉
|〈xy〉| le983155
〈xx〉983155
〈yy〉
The equality holds if and only if x and y are linearly dependent
Exercise Prove the inequality Hint write
x =〈xy〉〈yy〉y + z
Then 〈zy〉 = 0 Consider 〈xx〉Application For any hermitian positive definite matrix A
|ylowastAx|2 le (xlowastAx)(ylowastAy)
Numerical Linear Algebra Lecture 1 September 15 2020 5 17
3 Norm on a linear space V over a number field F (C or R)Definition A function 983042 middot 983042 V rarr R is called a norm if it satisfiesthe following three conditions (forall xy isin V and forall α isin F)(1) Nonnegativity
983042x983042 ge 0 983042x983042 = 0 hArr x = 0
(2) Positive homogeneity
983042αx983042 = |α|983042x983042
(3) Triangle inequality
983042x+ y983042 le 983042x983042+ 983042y983042
Exercise Show that a norm is a continuous function
Numerical Linear Algebra Lecture 1 September 15 2020 6 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Exercise Write the expression for v when the set q1q2 qnis only orthogonal
Cauchy-Schwarz inequality For any given inner product 〈middot middot〉
|〈xy〉| le983155
〈xx〉983155
〈yy〉
The equality holds if and only if x and y are linearly dependent
Exercise Prove the inequality Hint write
x =〈xy〉〈yy〉y + z
Then 〈zy〉 = 0 Consider 〈xx〉Application For any hermitian positive definite matrix A
|ylowastAx|2 le (xlowastAx)(ylowastAy)
Numerical Linear Algebra Lecture 1 September 15 2020 5 17
3 Norm on a linear space V over a number field F (C or R)Definition A function 983042 middot 983042 V rarr R is called a norm if it satisfiesthe following three conditions (forall xy isin V and forall α isin F)(1) Nonnegativity
983042x983042 ge 0 983042x983042 = 0 hArr x = 0
(2) Positive homogeneity
983042αx983042 = |α|983042x983042
(3) Triangle inequality
983042x+ y983042 le 983042x983042+ 983042y983042
Exercise Show that a norm is a continuous function
Numerical Linear Algebra Lecture 1 September 15 2020 6 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
3 Norm on a linear space V over a number field F (C or R)Definition A function 983042 middot 983042 V rarr R is called a norm if it satisfiesthe following three conditions (forall xy isin V and forall α isin F)(1) Nonnegativity
983042x983042 ge 0 983042x983042 = 0 hArr x = 0
(2) Positive homogeneity
983042αx983042 = |α|983042x983042
(3) Triangle inequality
983042x+ y983042 le 983042x983042+ 983042y983042
Exercise Show that a norm is a continuous function
Numerical Linear Algebra Lecture 1 September 15 2020 6 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Exercise For any given inner product 〈middot middot〉 let 983042 middot 983042 =983155
〈middot middot〉(1) Prove that the function 983042 middot 983042 =
983155〈middot middot〉 is a norm
(2) Prove the parallelogram law
983042u+ v9830422 + 983042uminus v9830422 = 2983042u9830422 + 2983042v9830422
(3) For a set of n orthogonal (with respect to the inner product〈middot middot〉) vectors xi prove that
983056983056983056983056983056
n983131
i=1
xi
983056983056983056983056983056
2
=
n983131
i=1
983042xi9830422
The function 983042 middot 983042 =983155
〈middot middot〉 is called the norm induced by the innerproduct 〈middot middot〉 Using this norm we can write the Cauchy-Schwarzinequality as
|〈xy〉| le 983042x983042983042y983042
Numerical Linear Algebra Lecture 1 September 15 2020 7 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Geometric interpretation of the standard inner product and itsinduced norm For simplicity we will consider vectors in R2 Let983042 middot 983042 be the norm induced by the standard inner product 〈middot middot〉 ie
983042 middot 983042 =983155
〈middot middot〉 then 〈uv〉 = 983042u983042983042v983042 cos θ
where 983042u983042 and 983042v983042 are the Euclidean length of u and v and θ(0 le θ le π) is the angle between u and v
Convergence of a sequence xk sub V xk rarr x
We say xk converges to x if limkrarrinfin 983042xk minus x983042 = 0
Numerical Linear Algebra Lecture 1 September 15 2020 8 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Equivalence of norms All norms on a finite-dimensional linearspace are equivalent in the following sense
Theorem 2
For each pair of norms 983042 middot 983042α and 983042 middot 983042β on a finite-dimensional linearspace V there exist positive constants a gt 0 and b gt 0 (depending onlyon the norms) such that
a le 983042x983042α983042x983042β
le b forall x( ∕= 0) isin V
Hint forallx =983123
i xivi where vi is a basis of V 983042x983042 =983123
i |xi| is anorm on V By 983042x983042α = 983042
983123i xivi983042α le 983042x983042 middotmaxi 983042vi983042α we know
983042 middot 983042α is a continuous function with respect to 983042 middot 983042 which attainsits minimum and maximum on the unit sphere x isin V 983042x983042 = 1(because it is a compact set) Then forallx isin V c le
983056983056983056 x983042x983042
983056983056983056αle C
Numerical Linear Algebra Lecture 1 September 15 2020 9 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
31 Vector norms on Cm
ℓp-norm
The norm 983042 middot 9830422 on Cm is also called the Euclidean norm which isinduced by the standard inner product on Cm
Matlab norm for 1 2infin norms
Numerical Linear Algebra Lecture 1 September 15 2020 10 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Weighted norm
Let 983042 middot 983042 denote any norm on Cm Suppose a diagonal matrixW = diagw1 middot middot middot wm wi ∕= 0 Then
983042x983042W = 983042Wx983042
is a norm called weighted norm For example weighted 2-norm
Dual norm
Let 983042 middot 983042 denote any norm on Cm The corresponding dual norm983042 middot 983042prime is defined by the formula
983042x983042prime = supyisinCm983042y983042=1
|ylowastx|
Numerical Linear Algebra Lecture 1 September 15 2020 11 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
32 Matrix norms on Cmtimesn
Frobenius norm forallA isin Cmtimesn define
983042A983042F =983059983131m
i=1
983131n
j=1|aij |2
98306012=
983059983131n
j=1983042aj98304222
98306012
or983042A983042F =
983155tr(AlowastA) =
983155tr(AAlowast)
Max norm983042A983042max = max
ij|aij |
Induced matrix norm (operator norm) forallA isin Cmtimesn define
983042A983042αβ = supxisinCnx ∕=0
983042Ax983042α983042x983042β
= supxisinCn
983042x983042β=1
983042Ax983042α = supxisinCn
983042x983042βle1
983042Ax983042α
where 983042 middot 983042α is a norm on Cm and 983042 middot 983042β is a norm on Cn We saythat 983042 middot 983042αβ is the matrix norm induced by 983042 middot 983042α and 983042 middot 983042β
Numerical Linear Algebra Lecture 1 September 15 2020 12 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Exercise forallx isin Cn prove that 983042Ax983042α le 983042A983042αβ983042x983042β Exercise Let A isin Cmtimesn B isin Cntimesr and let 983042 middot 983042α 983042 middot 983042β and 983042 middot 983042γbe norms on CmCn and Cr respectively Prove the inducedmatrix norms 983042 middot 983042αγ 983042 middot 983042αβ and 983042 middot 983042βγ satisfy
983042AB983042αγ le 983042A983042αβ983042B983042βγ
Exercise Prove that 983042A983042infin1 = maxij |aij | = 983042A983042max
The Frobenius norm 983042 middot 983042F on Cmtimesn is not induced by norms onCm and Cn See the following references
[1] Vijaya-Sekhar Chellaboina and Wassim M HaddadIs the Frobenius matrix norm inducedIEEE Trans Automat Control 40 (1995) no 12 2137ndash2139
[2] Djouadi Seddik MComment on ldquoIs the Frobenius matrix norm inducedrdquo With a reply byChellaboina and Haddad
IEEE Trans Automat Control 48 (2003) no 3 518ndash520
Numerical Linear Algebra Lecture 1 September 15 2020 13 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Induced matrix p-norm of A isin Cmtimesn For p isin [1+infin]
983042A983042p = 983042A983042pp = supxisinCn983042x983042p=1
983042Ax983042p
Example p-norm of a diagonal matrix D = diagd1 middot middot middot dm
983042D983042p = max1leilem
|di|
Example 1 2infin-norm
983042A9830421 = maxj
983131
i
|aij | 983042A983042infin = maxi
983131
j
|aij |
983042A9830422 =983155
λmax(AlowastA) =983155
λmax(AAlowast) le 983042A983042F
The norm 983042 middot 9830422 on Cmtimesn is also called the spectral norm
Matlab norm for 1 2infin-norm
Numerical Linear Algebra Lecture 1 September 15 2020 14 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
Example A =
9830631 20 2
983064
Numerical Linear Algebra Lecture 1 September 15 2020 15 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
33 Unitary invariance of 983042 middot 9830422 and 983042 middot 983042F forallA isin Cmtimesn
If P has orthonormal columns ie
P isin Cptimesm p ge m PlowastP = Im
then983042PA9830422 = 983042A9830422 983042PA983042F = 983042A983042F
If Q has orthonormal rows ie
Q isin Cntimesq n le q QQlowast = In
then983042AQ9830422 = 983042A9830422 983042AQ983042F = 983042A983042F
Numerical Linear Algebra Lecture 1 September 15 2020 16 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
4 Unitary matrix
For Q isin Cmtimesm if Qlowast = Qminus1 ie QlowastQ = I Q is called unitary(or orthogonal in the real case)
Exercise Let Q isin Cmtimesm be a unitary matrix Prove
983042Q9830422 = 1 983042Q983042F =radicm
A unitary matrix has both orthonormal rows and orthonormalcolumns
The columns of a unitary matrix form an orthonormal basis of Cm
Numerical Linear Algebra Lecture 1 September 15 2020 17 17
![Inner-Model Reflection Principles · 2020. 5. 6. · Inner-Model Reflection Principles 575 Friedman [4]. These meta-principles assert that certain sentences obtainable in outer models](https://static.fdocument.org/doc/165x107/60ca44a8ba91f907cd6b2d43/inner-model-reflection-principles-2020-5-6-inner-model-reiection-principles.jpg)
