Signal Processing 1 - nt.tuwien.ac.at · Resume Consider symmetric impulse response: h 0 =a, h 1...

Signal Processing 1

Signal Spaces

Univ.-Prof.,Dr.-Ing. Markus RuppWS 18/19

Th 14:00-15:30EI3A, Fr 8:45-10:15EI4

LVA 389.166

Last change: 22.11.2018

2Univ.-Prof. Dr.-Ing.

Markus Rupp

Resume Consider the amplitude function of a filter:

A(Ω)=1+cos(Ω)+2cos(2Ω) Question 1: is this a symmetric filter? Answer: potentially yes, but only if

dφ(Ω)/dΩ =const. Question 2: What is the impulse response

if this is a causal symmetric filter? Answer: h0=1; h1=0,5; h2=1; h3=0,5; h4=1.

or further delayed versions of this

Markus Rupp

Resume Consider symmetric impulse response: h0=a, h1=1, h2=a. Which values of |a|<0,5 deliver a linear phase filter?

h0+h1exp(-jΩ)+h2exp(-2jΩ) =exp(-jΩ)[h0exp(jΩ)+h1+h2exp(-jΩ)]

=exp(-jΩ)[a exp(jΩ)+1+a exp(-jΩ)] =exp(-jΩ) [1+2a cos(Ω)]

=exp(jφ(Ω)) A(Ω) φ(Ω)=-Ω

What happens for |a|>0,5? A(Ω)=|1+2a cos(Ω)|Thus, for each value of a from R we have a

linear phase system but the phase function may jump.

Markus Rupp

Resume Consider symmetric impulse response: h0=a, h1=1, h2=-a. For which values of a is this a linear phase system?

h0+h1exp(-jΩ)+h2exp(-2jΩ) =exp(-jΩ)[h0exp(jΩ)+h1+h2exp(-jΩ)]

=exp(-jΩ)[a exp(jΩ)+1-a exp(-jΩ)] =exp(-jΩ) [1+2aj sin(Ω)]

Thus, for every value of a from C with Real(a)=0, this is a linear phase system. Since we have:

exp(-jΩ) [1+2aj sin(Ω)] =exp(jφ(Ω)) A(Ω) φ(Ω)=-Ω

Markus Rupp

Resume Question: Is the sampling of a continuous function

a linear operation? Answer: Yes

[ ] )2()1()2()1(

)1()1(

)()()]([

ßfftßftfSftfS

Markus Rupp

Resume Question: Is the interpolation of sampled values a

linear operation? Answer: Yes

(1) (2) (1) (2)

(1) (2)

( ) ( )

k k k kk k

k kk k

I f T f p t kT f t

I f ßf T f p t kT T ßf p t kT

T f p t kT ß T f p t kT

f t f t

=−∞

∞ ∞

=−∞ =−∞

∞ ∞

=−∞ =−∞

= − =

+ = − + −

= − + −

∑ ∑

Markus Rupp

Resume Question: Is the conversion from fk to f(t)

and vice versa unique under equidistant sampling?

Answer: Yes

What is the answer for equidistant sampling at arbitrary start point f(kT+To)?

kTtpfTtf

−= ∑∞

−∞=

Markus Rupp

Resume Note: Many different sequences fk

(i) ;i=1,2,… can lead to the same time-continuous function f(t). However, always a different interpolator is required.

Example:

)2/();(

)2()1(

)2()2(

)1()1(

TkTffkTff

kTtpfT

kTtpfTtf

∑∞

−∞=

Resume Given f(kT)=fk, how do we get

gk=g(kT)=f(kT+T/2)?

Markus Rupp

+= ∑

−∞=

mTTkTT

fTkTfgm

mk 2sinc

Markus Rupp

Learning goals Vector Spaces and Applications of Linear Algebra

in Signal Processing (6Units, Chapter 2) Metric spaces, sequences, Cauchy-sequences, supremum,

infimum, sparsity (Ch 2.1) Groups, Vector spaces, linear combination, linear

independence, basis and dimension, orthogonality,blind channel estimation (Ch 2.2, 2.7)

Norms and normed vector spaces (Ch 2.3-2.5) Applications of norms: robustness descriptions,

feedback systems with nonlinear elements: the small gain theorem

Inner vector products and inner product spaces, Hilbert and Banach spaces (Ch 2.8-2.9)

Induced norms, Cauchy-Schwarz inequality: matched filter and correlation coefficient, time-frequency uncertainty (Ch 2.6)

Sets, Spaces, and Vectors A set is a collection of distinct objects. A vector is an n-dimensional set of n ordered

elements. A space is a set with “some added structure”.

Markus Rupp

Example 2.1: Metric

Code bookCode word

1,0==⊂∈

BYBCx n

Decoder has to decide which one among the received y is allowed and most probable!

Markus Rupp

Example 2.1: Metric

Code Rate:

)(log2 Mk =# Info bits

# Code bits kn ≥

nkR )(log2==

−),( kn Codexy

Markus Rupp

Example 2.1: Metric Decision of the „most probable symbol“

based on metric: Compare “distance” of received y to all possible,

allowed x. Symbol x with smallest (Hamming-) distance dH

is interpreted as correct.

( ) ∑ =⊕=

→×=n

Markus Rupp

Example 2.2 We transfer two possible signal forms from two

different sensors: f(t) and g(t). The receiver has to find out whether the

distorted received signal form r(t) is closer to f(t) or to g(t).

We need a distance measure from r(t) to f(t) and from r(t) to g(t):

∫∫ −=−=b

dttgtrgrdvsdttftrfrd 22

22 )()(),(.)()(),(

Markus Rupp

Metric Spaces Consider a set X in R,C,N,Z,B… Definition 2.1: A metric

is a functional mapping to measure distances between objects/elements of the set X. In order to call such a distance a metric, the following properties need to be satisfied:

)(: 00 NRXXd +→×

Xzyxzydyxdzxdyxyxd

xydyxdyxd

∈+≤==

,, allfor );,(),(),()3 ifonly and if ,0),()2

),(),()10),()0

Metric Note that 0) follows from 1), 2), and

3) since:

Markus Rupp

0),( :thus3) and 1) todue

),(2),(),(),(0:have wethen

)2 todue 0),(

=+≤=

yxdxydyxdxxd

Markus Rupp

Metric Spaces Example 2.3: Consider the following metric

d1(x,y)

defined over the vectors x,y. This metric is called l1 metric. (Manhattan distance)

Example 2.4: Consider the following metric dp(x,y)

( ) ∑ =

→×n

( ) ( ) pn

RRRd/1

Markus Rupp

Metric Spaces From the lp metrics the l1 and l2 metric as well as

the l∞ metric are of particular interest:

Example 2.5: Hamming Distance

( ) ( )iini

≤≤

=∞→∞

( ) ∑ =⊕=

→×=n

Markus Rupp

Metric Spaces Definition 2.2: A Metric Space (X,d) consists of

a set X and a metric d, valid on this set.

Example 2.6: The set Rn (vectors with n entries from R) together with the metric d2(x,y) builds a metric space.

Markus Rupp

Metric Spaces Note, a metric defines the distance to the zero

vector for p>=1:

A metric thus allows for statements about sizes (lengths,areas, volumes, etc for p=2) of objects. Tied to this is the existence of such objects.

Example 2.7: Consider the infinite long series of real or complex valued numbers xi, i=0,1,...,oo, with the property

∑∞

∞<0i

( ) ( ) ( ) pn

pip xxxd

100, ∑∑ ==

( ) ( ) ∞<= xdxd pp 0,

Markus Rupp

Metric Spaces Together with an lp metric, such sequences form a

metric space called lp(0,∞) space (Ger.: Folgenraum).

Consider on the other hand the sequence xi, i= -∞,...,-1, 0,1,...,∞ with the same properties and same metric, we have the lp(-∞,∞) space.

Consider the metric p ∞, for sequences for which: |xi| is bounded for every i, i.e., |xi|<M, we obtain the l∞(0,∞) or l∞(-∞,∞) space, respectively.

nnnnn yxyxd −=∞ sup),(

Markus Rupp

Supremum and Infimum Definition 2.3. (Supremum): Consider the set S in

R,Q,Z,N… with the elements xi. The smallest number z in R, for which we have:

is called supremum (sup) of the set S. It is the least upper bound.

If there is no number in R that is larger than the largest element in S, then we have sup(S)= ∞.

ixz i allfor ;≥

Markus Rupp

Supremum and Infimum Definition 2.4 (Infimum): Consider the set S in R

with the elements xi. The largest number z in R, for which we have:

is called Infimum (inf) of the set S. It is also called the greatest lower bound.

If there is no more number in R smaller than the smallest element in S, then we have inf(S)=- ∞.

ixz i allfor ;≤

Markus Rupp

Supremum and Infimum Example 2.8: Let S=(3,6) be an open set,

then: inf(S)=3, sup(S)=6

Let T=[4,8), then: inf(T)=4, sup(T)=8 Let U=[2,∞), then: inf(U)=2, sup(U)=∞

Why do we not simply select the maximum (minimum) of S?

Markus Rupp

Metric Spaces Metric spaces can also be formed with particular

properties:

Example 2.9: Let lh(0,∞) be a metric space of sequences in which sequences exist whose quadratic sum is finite (finite energy sequence). Thus, xi in lh(0,∞)=lp=2(0,∞) means, that

∞<∑∞

Markus Rupp

Metric Spaces Metric spaces can also be defined over

functions rather than a set of numbers (note that functions are in a way sets of numbers).

These are called metric spaces over functions (Ger.: Funktionenraum).

Markus Rupp

Metric Spaces Definition 2.5 (p-metric): Let X be a set of real-

or complex-valued functions, defined on the interval [a,b] with b>a, with p>=1, that have the property:

The metric over the functions x(t),y(t) from X is then:

(Lebesgue Integral)

∞<≤

−= ∫ pdttytxyxd

pp 1;)()(),(

∞<≤∞<∫ pdttxb

p 1;)(

Markus Rupp

Metric Spaces The space with the metric dp over the functions is

called Lp space.

For p∞ we have for bounded functions (from above and below) :

For more details on Lp, read Ch 2.1.3

btatytxyxd

≤≤−=

;)()(sup),(

)(sup ],[

Markus Rupp

0 1 2 3 4 5 6 7 8 9 10

)()(),(

txtxtx

>>>><<

),(),(),(),(

xxdxxdxxdxxd

∞ ),(),(2

xxdxxd

Metric: Problems with Pytagoras?

Markus Rupp

Under which dp metric is the picture correct?

p-Metrics: consider iso-metricἴ iσος=equal

Markus Rupp

p=2p=4 p=100

inflation

deflation( )

Metrics for 0<p<1 What happens if we select 0<p<1? In this case

does not satisfy the triangle equality (subadditive property)

Markus Rupp

( ) ( ) pn

piip yxyxd

1, ∑ =

Metrics for 0<p<1 Example 2.10: select the three

twodimensional points x=(1,0),y=(0,1) and z=(0.5,0.3)

compute

Markus Rupp

2 21 12 22 2

1.57 2.38

n ni i i ii i

x z z y

> − + − =

∑ ∑

Metrics for 0<p<1 Definition 2.6: For 0<p<1, the

following expression is a metric:

Example 2.10 again:

Markus Rupp

( ) ( )∑ =−=

piip yxyxd

( )122

1 12 22 2

1.25 1.54

ni iip

n ni i i ii i

d x y x y

x z z y

= − =

< − + − =

∑ ∑

d0-Metric Is there also a d0 metric? Answer: yes but the definition

causes some mathematical difficulties as discontinuities occur for

Markus Rupp

( ) ( )∑ =→ −=n

piip yxyxd

100 lim,

=elsex

Sparseness Definition 2.7: In practise this definition

of d0 is often given in the form of

d0(x)=Σxi0

which is truly a counter for sparseness. But note, this is not a metric (norm)

sometimes it is called a pseudonorm

Markus Rupp

Example 2.11Reconsider Sampling For bandlimited signals we found:

Given the interpolation function, what is the optimal sequence gk to minimize

Answer: gk=fk=f(kT)min=038

Univ.-Prof. Dr.-Ing. Markus Rupp

−∞=

)(sinc),(min

)(sinc)(

2 kTtT

)(sinc)(min

−−= ∫ ∑

∞−

−∞=

dtkTtT

Example 2.11Reconsider Sampling Given a set of interpolation functions pm(t) :

Find the pair (gk,pm(t));m=1,2,..,M that results in a sparse representation of f(t)!

basic concept for modern speech coding (CELP)

Markus Rupp

TktkTtpgtf

Mmmkgk

,...,2,10

0);()(

≤≤−= ∑=

that such

CELPCodebook excited linear prediction We can alternatively also select from a set of

excitation sequences g1,k,g2,k,…,gM,k. The interpolator can be constructed in form of a

fixed part with a linear filter (predictor) before:

The task now is to find the following minimum:

Markus Rupp

−= ∑

)(sinc)(1

−+ ∑

,2 )(),(minmink

kkmDpg kTtpgttfd

prediction for tD>0

CELPCodebook excited linear prediction

Problem is split into a search over a fixed set (gm,k) and the minimization of a d2-metricLeast Squares (Chapter 3)

Markus Rupp

)(sinc)(minmin

)()(minmin

)(),(minmin

−−−+=

−−+=

∫ ∑ ∑

∫ ∑

l lkmDpg

kkmDpg

dtkTlTtT

dtkTtpgttf

kTtpgttfd

Markus Rupp

Metric Spaces lp(0,∞) : Space of causal sequences lh(0,∞) : Space of causal sequences with

finite energy (p=2) lp(-∞,∞): Space of non-causal sequences

(this includes lp(0,∞)) Lp(0,∞): Space of causal functions Lp(-∞,∞): Space of non-causal functions (C[a,b],dp): Space of continuous functions

Markus Rupp

Metric Spaces Example 2.12 Filter design: a linear time-discrete filter with linear

phase H(exp(jΩ)) =a0+a1exp(jΩ)+a1exp(-jΩ)=a0+2a1cos(Ω) is to design, so that it follows a desired amplitude function |Hd(exp(jΩ))|=1 for |Ω|< ΩG (zero else) most optimally.

( ) ( )

[ ] [ ]

)(;)sin(;

)sin(42)1(min

)cos(2021)cos(21

)cos(221min

series Fourier cmp.

daadaa

Ω−++−−Ω

ΩΩ−−+ΩΩ−−=

ΩΩ−−=

∫∫

Markus Rupp

Minimization ofquadratic metric d2

Example 2.13. Recall Paley Wiener Given an amplitude function Ad(Ω), what is

a valid filter function F(ejΩ), i.e.Ad

2(Ω) =|F(ejΩ)|2

Different to the previous example the full information of the desired Filter is not known, only the amplitude function is given.

We recall that linear phase filters are well described by their amplitude function.

Markus Rupp

Example 2.13. Recall Paley Wiener Linear phase filters have symmetric

impulse responses

Markus Rupp

Example 2.13. Recall Paley Wiener Problem formulation:

Recall that:

Markus Rupp

Example 2.13. Recall Paley Wiener For which we find:

For l=0,1,…,N. From there we can compute the original coefficients fn.

Thus, given Ad, we now know how to compute F.

Markus Rupp

Resume What is a set? What is a vector? What is a space? What is a metric space?

Markus Rupp

Resume Definition: Space X Objects (numbers,

sequences, functions…) with structure Definition: Metric space (X,d) space X on

a metric d is defined as

Xzyxzydyxdzxdyxyxd

xydyxdyxd

∈+≤==

,, allfor );,(),(),()3 ifonly and if ,0),()2

),(),()10),()0

Resume When do sequences in metric spaces

with lp metric exist?

Markus Rupp

∑∞

∞<0i

Markus Rupp

Resume lp(0,∞) :

Metric space of causal sequences with p metric

lh(-∞,∞): Metric space of non-causal sequences with 2 metric

Markus Rupp

Metric Spaces Robustness (Ger.: Robustheit):

Often it is important not to optimize the mean value (e.g., mean data rate, BER) but the value at worst case condition.

If a systems shows to be insensitive to the worst cases, it is called robust.

Markus Rupp

Metric Spaces Robustness is often measured in terms of energy Let xi be the input sequence and yi the output sequence of a

system T. The output is assumed to be distorted by an unknown sequence vi. The form of distortion is not necessarily additive or even known. Furthermore, the M initial states zi of this system are also not known. Assume that there exists a reference system without distortion having an output signal yi

(R). The influence of such distortion can be described be the

following expression:( )

γ≤+

∑∑

Markus Rupp

Metric Spaces Or somewhat more precise:

Assume now that there exists several possibilities for the realization of this system T(F) depending on a specific strategy. Then the robustness criterion reads:

),0(),,1(sup γ≤

∑∑

∈∈N

NlvMlz vz

∑∑

∈∈ +

NlvMlzFvz

),0(),,1(supinf

Markus Rupp

Example 2.19: Robustness A power amplifier PA in a cell phone (Ger.: Handy)

can be described by the following nonlinear mapping:

The variable ρ= ρi(T) describes the influence of temperature, aging and more.

In order to define the robustness of the output signal with respect to ρ, the following expression is considered:

iii vx

),0(,sup γ

ρρ≤

∑∑

PA yiT

Markus Rupp

Example 2.19: Robustness If it is possible to reduce the influence of

ρ by new technologies or improved circuitry, a smaller factor γ is the result:

The improvement is achieved by higher robustness (smaller γ).

Note: Robustness is not sensitivity.

),0(,sup γγ

ρρ<≤

∑∑

∈newN

PA yiT

Contr.

Markus Rupp

Metric Spaces Convergence of sequences

Definition 2.8 (limit): If there exists to every distance δ a number no so that d(xn,y)< δ for each n>no at fixed value y, the sequence xn is said to be convergent to y.

y is called the limit (Ger.: Grenzwert) of xn. All points in an arbitrarily small distance to y are called the

neighborhood (Ger.: Nachbarschaft) of y.

∞→=→lim

Markus Rupp

Metric Spaces Example 2.14: the following two sequences

diverge:

Definition 2.9: If a sequence xn returns infinitely often to a neighborhood of z, then we call this point z a limit point (Ger.: Häufungspunkt, Grenzpunkt). E.g., the sequence bn takes on the limit points z=0 and z=2.

If limit points exist, then there must be subsequences (partial sequences, Ger.: Teilfolgen) xn that converge.

Markus Rupp

Metric Spaces The largest limit point of a sequence xn is called

limes superior, or

The smallest limit point of a sequence xn is called limes inferior, or

A sequence converges, if:

nn x∞→suplim

nn x∞→inflim

nnnn xx ∞→∞→ = supliminflim

Markus Rupp

Metric Spaces Example 2.15: Consider the sequence

There are two limit points. The subsequence c0,c2,c4,.. takes on the limit 4, while the subsequence c1,c3,c5,.. takes on the limit 0.

12 2 ( 1) ; 1,2,3,...

limsup 4 but : sup 4,5liminf 0

n n n n

→∞

= + + ⋅ − =

Markus Rupp

Metric Spaces Definition 2.10: A sequence xn in a metric

space (X,d) is called Cauchy-sequence, if there exists for every ε>0 a N(ε)>0 so that d(xn,xm)<ε for every m,n>N.

It is possible to prove that sequences that converge are Cauchy-sequences. The opposite is not always true.

Markus Rupp

Metric Spaces Example 2.16: Let X=C[-1,1] be a set of

continuous functions and fn(t) a sequence of functions, defined by

Consider the metric space (X,d2) with

>≤≤−+

ntntnnt

/11/1/12/12/

( )∫−

22 )()(),( dttgtfgfd

Markus Rupp

Metric Spaces

nn /1/1

Example 2.16:

Markus Rupp

Metric Spaces We find

With d20 for large m and n, m-n=k<oo. We thus conclude that it is a Cauchy sequence.

Note, however, the function in the limit is:

This function is not continuous and thus not in X=C[-1,1]. The sequence is thus not convergent in X!

tftfd mn

))(),((

=0102/100

Markus Rupp

Metric Spaces The reason why the Cauchy-sequence does

not converge in X, can be interpreted as a hole in the set of functions X.

If we had defined X as the set that included discontinuous functions as well, then we had obtained a Cauchy sequence that converges.

Definition 2.11: A metric space (X,d) is called complete (Ger.: vollständig), if every Cauchy-sequence converges in X.

Markus Rupp

Metric Spaces Gibbs‘ Phenomenon You are probably well aware of the fact that

periodic functions can be described by Fourier series.

Example 2.17: Consider the following periodic signal of period 1 in L[-0,5 , 0,5]:

with the Fourier-series:

<<≤≤−−<≤−

=5,025,0025,025,0125,05,00

−−

−+= ∑

n ππ

)12(2cos12

)1(25,0)(1

Josiah Willard Gibbs (11.2.1839 –28.4.1903) was an American scientistin Physics, Chemistry, Mathematics

Markus Rupp

Metric Spaces Let‘s run this to noo Note that in this case we have:

0))(),((lim0))(),((lim 2

∞∞→

∞→

tftfdtftfd

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5-0.2

)(500 tfn=

Example 2.18 Let us consider the problem of finding a

square root of a number A (e.g. A=3 in N) We thus want to find a solution for x2-A=0 An iterative algorithm (Heron) can be

derived (Newton-style):

Markus Rupp

Heron from Alexandria (Mechanicus; † > 62) Greek mathematician and engineerMethod is known since 1700 BC.

Example 2.18 Obviously, starting with a value from

Q, results in a new value from Q:

For noo, we obtain a value from R!

Markus Rupp

Rules in the Vector Space Obviously we want to „do“ something

with the objects in our set(space) We can stretch them

We can add them

Markus Rupp

x xα→

,x y x y→ +

Binary operation Definition 2.12: A binary operation *

on a set S is a rule that assigns to each ordered pair (a,b) of elements from S some element from S. Since the operation ends with an element in the same set, we call this also a closed operation.

Example 2.20: S=Z, *: a*b=min(a,b)

Markus Rupp

Vector Spaces Definition 2.13: A linear vector space S over a set of

scalars T (C,R,Q,Z,N,B) is a set of (objects) vectors together with a binary additive „+“ and a scalar multiplicative „.“ operation, satisfying the following properties:

1) S is a group under addition. 2)

3) W.r.t. the multiplicative operation there exists an Identity (One) and a Zero element:

yaxayxaxbxaxbaxabxbaSxa

SyxTba

+=++=+=∈

)()(,)()(,

:,, have we in and everyFor

00;1 =•=• xxxIdentity elementw.r.t „+“!

Markus Rupp

Groups Definition 2.14 (Group): A set S for which a binary

operation * (operation w.r.t two elements of S) is defined, is called a group if the following holds: 1) for each a,b in S it holds that: (a*b) in S 2) there exists an identity element e in S, so that for every

element a in S: e*a=a*e=a. 3) for each element a in S there exists an inverse element b in

S, so that: a*b=b*a=e. 4) The binary operation * is associative, i.e.,: (a*b)*c=a*(b*c)

The group is denominated by (S, *). If, furthermore, for each pair a,b in S it holds that a*b=b*a

(commutativity), then the group is called commutative or Abelian (Ger.: Abelsch).

Markus Rupp

Groups Example 2.21: Let S be the set of vectors of a

particular dimension. S forms a group w.r.t the additive operator, if the following properties are satisfied:

( ) ( )zyxzyxSzyx

SySxxxx

SyxSyx

++=++∈

−==+

∈∈=+=+

∈+∈

:that holdsit eachFor :eassociativ is operation additive The

that so , second a exists there element eachFor

that so , in element identity an exists There

:that holdsit everyFor

Markus Rupp

Rings Definition 2.15 (Ring): A set S for which two

binary operations + and * are defined, is called a ring if the following holds: (S,+) is a commutative group (Abelian) The operation * is associative. Distributivity holds w.r.t. + for all scalars:

a(b+c)=ab+ac, (a+b)c=ac+bc. A ring is denominated by (S,+, *). Note: for * there does not need to be an identity

or inverse element. If there exists additionally the inverse element to

*, then it is called skew field or division ring (Ger.: Schiefkörper).

Markus Rupp

Fields Definition 2.16 (Field): A set S equipped

with two binary operations + and * is called a field (Ger.: Körper), if: 1) (S,+) is an Abelian group 2) (S\0,*) is an Abelian group 3) The operations + and * distribute.

If the set is finite, i.e., |S|<oo, we talk about finite groups, finite rings, finite fields….

Markus Rupp

Fields Example 2.22: The Galois Field GF(2) is a

field. The elements of the field are 0 and 1. Define the operations + and *

a b a+b0 0 00 1 11 0 11 1 0

a b a*b0 0 00 1 01 0 01 1 1

Markus Rupp

Fields W.r.t. + S needs to be an Abelian Group:

a+b is in S. Identity element is 0: 0+0=0, 1+0=1. Inverse element exists: 0+0=0,1+1=0. Associativity: (a+b)+c=a+(b+c). Check the remaining properties yourself...

Markus Rupp

Fields Example 2.23:

(R,+,*), (Q,+,*),(C,+,*) are fields; (N,+,*),(Z,+,*) not!

Markus Rupp

Relation to each other

Ring AbelianGroup

Vector space

SkewfieldField

Markus Rupp

Vector Spaces Example 2.24: for finite dimensional linear vector

spaces: 1) Consider the linear vector space in R4 (set of

quadruples, Ger.: Menge der Quadrupel)

2) The set of m X n matrices with real-valued elements.

3) The set of polynomials of degree 0..n with real-valued coefficients.

2121913

yxyxyx

Markus Rupp

Vector Spaces Still Example 2.24: Consider the (3,2)

single parity check code in GF(2)3 with the elements:V=[000],[011],[101],[110]

„+“ now means „exclusive or“

Do the remaining elements W=[111],[100],[010],[001] also form a linear vector space?

Markus Rupp

Vector Spaces Example 2.25: for infinite-dimensional linear

vector spaces:

1) Consider the set of infinitely long sequences

2) The set of continuous functions over the

interval [a,b] C[a,b]

3) The set of functions in LpLp[a,b]

Markus Rupp

Vector Spaces Definition 2.17: Let S be a vector space. If V is a

subset of S such that V itself is a vector space then V is called a subspace (Ger.: Unterraum) of S.

Example 2.26: Let S be the set of polynomials of arbitrary degree (>6) and V the set of polynomials of degree less than 6. Then V is a subspace of S.

Example 2.27: A (n,k) binary linear block code is a k-dimensional subspace of GF(2)n.

Markus Rupp

Vector Spaces Definition 2.18: Let S be a linear vector space

over R and T a subset of S. A point x of S is called linear combination (Ger.: Linearkombination) of points in T, if there is a set of points pi ;i=1,2,...,m in T and a finite set of scalars ci ;i=1,2,...,m in R, so that:

Note that the set T does not need to be finite.

mm pcpcpcx +++= ...2211

Markus Rupp

Vector Spaces Example 2.28: Let S=C(R), the set of continuous

functions over the complex (real) numbers. Let, furthermore, p1(t)=1; p2(t)=t, p3(t)=t2. A linear combination of such functions is given by:

Consider the polynomial x(t)=-1+5t+t2, and the function p4(t)=t2-1.

Obviously, the description is not unique. The number of required coefficients varies.

2321)( tctcctx ++=

)()(5)(2)()(5)(

)()(5)()(

tptptptptptp

tptptptx

+=+−+=

++−=

Markus Rupp

Vector Spaces Definition 2.19: Let S be a linear vector space

and T a subset of S. The subset T (pi ;i=1,2,…,m) is said to be linearly independent (Ger.: linear unabhängig), if for each nonempty linear subset T the only finite set of scalars satisfying the equation

is the trivial solution c1= c2 =...= cm =0.

If the above equation is satisfied by a set of scalars that are not all equal to zero, then the subset (pi ;i=1,2,...,m) is called linearly dependent(Ger.: linear abhängig).

0...2211 =+++

mm pcpcpc

Markus Rupp

Vector Spaces Example 2.29: The previously presented

polynomials p1(t)=1; p2(t)=t, p3(t)=t2, p4(t)=t2-1 are linearly dependent, since:

The polynomials p1(t),..., p3(t) are linearly independent!

The vectors p1=[2,-3,4],p2=[-1,6,-2] and p3=[1,6,2] are linearly dependent since:

0)()()( 431 =+− tptptp

0354321

=++ ppp

Markus Rupp

Vector Spaces Example 2.30: Consider the complex numbers

z=r+j i in R2 in vector form:

This is a set describing z and z*. Note that both elements are linearly independent as long as i is unequal to zero!

Markus Rupp

Vector Spaces Example 2.31: Consider band-limited functions or

sequences xk (also band-limited random processes) that only exist in a frequency range S. For a sequence fk (linear time-invariant system) existing in the complementary space of S we find:

Band-limited signals are said to be linearly dependent! Note however, that the property “finite number of

elements” is not necessarily satisfied.

( ) ( )( ) ( ) ( )

eXeFeYf

SeFSeX

−∞=−

ΩΩΩ−

−∞=

∑∑

−−=

==⇔==

∈Ω=∉Ω=

for for with xLet

Markus Rupp

Example 2.33 Consider blind channel estimation (for

equalization):

Markus Rupp

Example 2.33 Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

We obtain the equation:

Which properties must the receive vectors rk(1) and rk

have, so that the problem can be solved uniquely? Vectors rk

(1,2) need to be linearly dependent!

[ ] 0,...,,,,...,,

......

)2()2(2

)1()1(2

)2(222

)2(121

)1(212

)1(111

+++=+++

rrrrrr

rgrgrgrgrgrg

Markus Rupp

Vector Spaces At this point several interesting questions arise:

Under which conditions is the linear combination of vectors unique?

Which is the smallest set of vectors required to describe every vector in S by a linear combination?

If a vector x can be described by a linear combination of pi ;i=1..m, how do we get the linear weights ci ;i=1..m?

Of which form do the vectors pi ;i=1..m need to be in order to reach every point x in S?

If x cannot be described exactly by a linear combination of pi ;i=1..m, how can it be approximated in the best way (smallest error)?

Markus Rupp

Resume Space Metric complete

Convergence Cauchy-Sequence

Set vs. Vector

Markus Rupp

Resume Consider the following sequences in the

metric space l∞(1,∞):

Consider a classical Gaussian noise sequence from N(0,σ2). Is such a sequence in the metric space lp(-∞,∞)?

≥−>

∞ nyxyxd

allfor ;that Proof

Markus Rupp

Resume What can be said about the convergence of

these causal sequences (n=1,2,..)?

Which of these sequences is in the metric space l2(1,∞), which in l∞(1,∞)?

)1(111000

)1(111

−−

−−+=

Markus Rupp

Resume Consider the following sequence in Q in the

metric space (Q,d∞):

? in sequence-Cauchya thisIs

converge? sequence thisDoes

,...100000141421,

1000014142,

10001414,

100141,

Not in Q

Resume Problems in Q: Consider Is this convergent in Q? Similarly, consider

Markus Rupp

nm ∑

nm =∑

=∞→ !

π=∑

=∞→ n

Markus Rupp

Resume Which of the following sets of vectors are LI and which not?

Markus Rupp

Resume Given the set pi and a vector x, how do we obtain the linear

weights?1 21 2

... m mp p p xα α α+ + + =

1 2 1 2 1 2

, ,...,

, ,..., , ,..., , ,...,

p p p x

p p p p p p p p p x

Markus Rupp

Resume: LI Consequence of missing LI in linear systems of

equations:Rows < Columns

equations of system thefor solution a be Let 00

due to linear dependency in the column vectors, a 0 must exist so that0

Then we must also have that: 0 .Thus, an infinite amount of solut

x A xa a

Ax A x x b bα

= + = + =

ions exists for this system of equations.

Markus Rupp

System Identification Problem: The impulse response hk of a

linear time-invariant system is to estimate based on observations of input and output signals. What conditions does the input skhave to satisfy to ensure a uniqueidentification?

hsk rk

Markus Rupp

System Identification Solution:

11 2 1 1

hs s s s rh

−−

Markus Rupp

11 2 1 1

22 3 1 2

hs s s s rhs s s s r

−−

Markus Rupp

For a unique identification, the rows (columns) need to be linearly independent!

11 2 1 1

22 3 1 2

01 2 2 2 1

M M M M M

hs s s s rhs s s s r

hs rhs s s s r

−−

− −

+ − −

Markus Rupp

System Identification A signal with such a property is said

to be of persistent excitation (Ger.: hartnäckige Anregung).

Question: is a complex-valued exponential harmonic sk=exp(jΩοk) of persistent excitation?

What about a sinusoid sk=sin(Ωοk)?

Persistent Excitation Consider excitation signal

only system of order 1 (constant) can be identified.

[ ][ ] 1

)1(201

,...,,,,...,,,

xeeeeexeeeex

jMjjjj

ΩΩΩΩΩ

−ΩΩΩΩ

Univ.-Prof. Dr.-Ing. Markus Rupp111

Persistent Excitation Consider

excitation signal:

only system of order 2 can be identified.

( )[ ][ ]

jkjjkjok

Mjjjjb

Mjjjja

kjkjok

eeeexeeeex

21))1(sin(

,...,,,,...,,,

21)sin(

Ω−Ω

Ω−Ω−ΩΩ+

−Ω−Ω−Ω−Ω−

−ΩΩΩΩ

Ω−Ω

−=+Ω=

−=Ω=

Persistent Excitation How many signals (frequencies Ωo) of the

form exp(jΩok) are required to identify a time-invariant system of order M?

How many signals (frequencies Ωo) of the form sin(Ωok) are required to identify a time-invariant system of order M?

Markus Rupp

Vector Spaces Definition 2.20: Let T be a set of vectors in a

vector space S over a set of scalars R(C,Q,Z,N,B). The set of vectors V that can be reached by all possible (finite) linear combinations of vectors in T is the span (Ger.: aufgespannte Menge, erzeugte Menge, lineare Hülle) of the vectors. :

)span(TV =

Vector Spaces Note that this is an elegant way of saying something in short

that otherwise would require a lengthy formulation. If T is a set, then span(T) is a set as well!

The number field is typically K=R or K=C. The span is typically defining a vector space!

Markus Rupp

∈∈==

∈+==

,|)span(

,....,2,1|,|)span(

TxKxTV

NixTKyxTV

βαβα

Markus Rupp

Vector Spaces Definition 2.21: Let S be a vector space, and let

T be a set of vectors from S such that span(T)=S. If the elements in T are linearly independent, then T is said to be a Hamel basis for S.

Example 2.34: The vectors p1=[1,6,5], p2=[-2,4,2], p3=[1,1,0] and p4=[7,5,2] are linearly dependent. Note that T=p1,p2,p3 spans the space R3 and thus is a basis for R3.

Example 2.35: The vectors p1=[1,0,0], p2=[0,1,0] and p3=[0,0,1] are linearly independent and are a basis for R3. This basis is often called natural basis (Ger.: natürliche Basis).

Georg Karl Wilhelm Hamel (12 September 1877–4 October 1954)

Markus Rupp

Vector Spaces Example 2.36: Consider the (3,2) single parity

check code in GF(2)3 with the elements:V=[000],[011],[101],[110].A Hamel basis is given by:

=]011[]101[

Markus Rupp

Example 2.37 The following 3x3 matrices are a Hamel basis in R3x3:

100000000

,010000000

,001000000

000100000

,000010000

,000001000

000000100

,000000010

,000000001

Markus Rupp

Example 2.38 Are these 3x3 matrices a Hamel basis?:

Answer: These matrices are not a basis for R3x3. However, the are a basis for the subspace in R3x3,

that has a zero row and column sum!

−−

110110000

,101101000

,110000110

,101000101

Markus Rupp

Vector Spaces Definition 2.22: The number of elements in a set

is its cardinality |A| (Ger.: Kardinalität).

Theorem 2.1: If two sets T and U are Hamel bases for the same vector space S, then T and U are of the same cardinality.

Proof: (only for the finite dimensional case). Letbe two bases for S and with at least one coefficient unequal to zero,

,...,,;,...,,2121 nm

qqqUpppT ==

mm pcpcpcq +++= ...22111

4,1,1,1,1 =−−+−−+= AA :QPSK jjjj

Georg Cantor (3.3.1845-6.1.1918), German Mathematician

Markus Rupp

Vector Spaces say c1, then: Thus

is also a basis for S. Further substitution leads to

( ),...,,

pcpcqc

p −−−=

,,...,,,

,,...,,,...

,...,,,

Markus Rupp

Vector Spaces Consequently, we must have: Now starting with q1 instead of p1, we

obtain Thus, we must have n=m.

Definition 2.23: Let T be a Hamel basis for S. The cardinality of T is the dimension of S, |T|=dim(S). It equals the number of linearly independent vectors, required to span the space S.

nm ≥

.mn ≥

Markus Rupp

Vector Spaces Note: each vector space has at least one

Hamel basis.

Operations in the vector space are often simpler and of lower complexity in their basis.

Markus Rupp

Example 2.33 revisited Consider blind channel estimator (blind

equalization):

Markus Rupp

Example 2.33 revisited Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

When are the vectors linearly dependent?

In order to obtain linear dependency, we must have a sufficient number of vectors r1

(1)..rm(1), r1

(2)..rm(2), so that

2m>N since the vectors rk(1,2) of dimension N can maximally

span RN!

[ ] mN

rrrrrr

rgrgrgrgrgrg

)2()2(2

)1()1(2

)2(222

)2(121

)1(212

)1(111

R;0R,...,,,,...,,

......

×∈==

+++=+++

Markus Rupp

Inner Products Definition 2.24: A vector space for which an

inner vector product is defined is said to be a pre- Hilbert space (Ger.: Innerer Vektorproduktraum, Vor-Hilbertraum).

An inner vector product maps two vectors onto one scalar with the following properties:

zyzxzyx

else. 0 and 0for 0,)0*

Markus Rupp

Inner Products Example 2.39: Consider the space of continuous functions

C[a,b] with the two elements x(t) and h(t). Let x(t) be an input signal and h(t) the impulse response of a low pass. We have:

Example 2.40: Consider vectors in C: Matrices can also build inner products:

dThxTy

)()()(

τττ

BAABH ,)Trace( =

xyyx H=,

Markus Rupp

Inner Products Example 2.41: Consider the expectation of a

random variable:

This is also an inner vector product, however with an additional weighting function.

dxdyyxfxy

),(xyE xy

= ∫∫

Inner Products Consider two 2-dimensional vectors x and y. Their inner product (projection) is a

measure of non-orthogonality:

Markus Rupp

000 <=> y,xy,xy,xy

Markus Rupp

Complete Space Remember:

Definition 2.11: A metric Space (X,d) is called complete (Ger.: vollständig), if every Cauchy-sequence converges.

Markus Rupp

Hilbert- and Banach Spaces Definition 2.25: A complete normed

(metric) vector space is said to be a Banach space. Is there additionally an inner vector product (the norm is an induced norm), the space is said to be a Hilbert space.

Example 2.42: The space of continuous functions (C[a,b],d∞) is a Banach space.

Example 2.43: The space of continuous functions (C[a,b],dp) is for finite p not a Banach space since it is not complete.

Stefan Banach, (30.3.1892-31.8.1945) Polish Mathematician

Markus Rupp

Hilbert- and Banach Spaces Example 2.44: The space of sequences lp(0,∞) is a

Banach space. For p=2 it is also a Hilbert space. Example 2.45: The space of functions Lp[a,b] is a

Banach space. For p=2 it is also a Hilbert space.This Hilbert space is often denoted as L2(R) for functions and l2(R) for sequences.

In the following part of the lecture we will exclusively consider Hilbert spaces. (if not noticed otherwise)

Markus Rupp

Orthogonality Definition 2.26: Vectors of a pre-Hilbert space

are said to be orthogonal or perpendicular (Ger.: normal) if <x,y>=0.

Definition 2.27: Vectors of a pre-Hilbert space are said to be orthonormal if:

Markus Rupp

Orthogonality Example 2.46: The following set of

vectors is orthogonal:

How do we have to modify the set, in order to make the vectors orthonormal?

−−

Orthogonality Consider a set of orthogonal vectors.

We then have:

Consider a set of orthonormal vectors. We then have:

Markus Rupp

iii pppp ,,

11∑∑∑

= ααα

∑∑∑===

iii pp

11, ααα

Markus Rupp

Orthogonality Example 2.47: The following set of functions is

orthonormal in [-π, π]:1,ejt/4, ej2t/4, ej3t/4

The inner product is defined as:

0)4/exp(21

)4/2exp()4/exp(21,

:)4/2exp()( and )4/exp()(Let

)()(21, *

dtjtjtgf

jttgjttf

dttgtfgf

Markus Rupp

Vector Spaces At this point several interesting questions arise:

Under which conditions is the linear combination of vectors unique?

Which is the smallest set of vectors required to describe every vector in S by a linear combination?

If a vector x can be described by a linear combination of pi ;i=1..m, how do we get the linear weights ci ;i=1..m?

Of which form do the vectors pi ;i=1..m need to be in order to reach every point x in S?

If x cannot be described exactly by a linear combination of pi ;i=1..m, how can it be approximated in the best way (smallest error)?

Markus Rupp

Vector Spaces Definition 2.28: A Hamel basis of dimension m is

said to be orthogonal if for all basis vectors T=p1,p2,p3...,pm:

Definition 2.29: A Hamel basis of dimension m is said to be orthonormal if all basis vectors T=p1,p2,p3...,pm:

=≠≠

ppji ;0

ji jiji

δ;1;0

Markus Rupp

Orthogonality Let T=p1,p2,...,pn be a set of vectors. How can we

find a set S=q1,q2,...,qm with m smaller or equal to n so that span(S)=span(T) and the vectors in S are orthonormal?

Gram-Schmidt Method: 1) take p1 and construct

2) build:

3) continue:

4) if ei=0, throw away pi+1.

Erhard Schmidt 13.1.1876- 6.12.1959 German Mathematician

22311332

111211221

qqpqqppe

eeeqqqppe

−−=

Markus Rupp

Orthogonality Advantages of orthogonal bases:

[ ] . of tionmultiplicaleft by Proof

basis lorthonorma an be let

ppfppfppff

pppPcbaf

332211

,,],,,[

Markus Rupp

Vector Spaces Definition 2.30: If there are two bases,

that span the same space with the additional property:

then these bases are said to be dual or biorthogonal (biorthonormal for ki,j=1).

,...,,;,...,,2121 mm

qqqUpppT ==

jijijikqp −= δ,,

Markus Rupp

Vector Spaces Example 2.48: let:

These pairs build a dual basis in R2. Consider

[ ] [ ] [ ] [ ] ;1,0;1,1;1,1;0,12121

TTTT qqpp ==−==

Tbaf ],[=

212211

qbaqaqpfqpff

pbpbapqfpqff

+−+=+=

Markus Rupp

Resume Space Metric Set vs. Vector complete

Convergence Cauchy-Sequence

Linear vector space Subspace

Markus Rupp

Resume Consider the following function

in C[-∞,∞]:[ ]

( )?,C[- in converge it Doesto? converge sequence this does Where

: for sequence functional the Compute

∞∞

−−=

σπσ

( )?,L[- in converge it Does pd],∞∞

-50 -40 -30 -20 -10 0 10 20 30 40 50

Markus Rupp

Resume Cardinality of a set of QPSK?

Dimension of a vector space?= cardinality of the Hamel basis

What is the dimension of the vectors that form a Hamel basis for QPSK ?

What is a linear combination? When are vectors linearly independent?

4,1,1,1,1 =−−+−−+= AA :QPSK jjjj

Markus Rupp

Resume00 0

1 01 1

1 02 2

1 1 01 1

......

hy xh hy x

h hy xh h h

xh h hy x

h h hy x

y h hh h

y hy hy h

−− −

− + −

− −

1 0...

h xy H x

− − +

Which column vectors of these matrices are LI?

Markus Rupp

Example 2.33 revisited Consider blind channel estimator (blind

equalization):

Markus Rupp

Example 2.33 revisited Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

When are the vectors linearly dependent?

In order to obtain linear dependency, we must have a sufficient number of vectors r1

(1)..rm(1), r1

(2)..rm(2), so that

2m>N since the vectors rk(1,2) of dimension N can maximally

span RN!

[ ] mN

rrrrrr

rgrgrgrgrgrg

)2()2(2

)1()1(2

)2(222

)2(121

)1(212

)1(111

R;0R,...,,,,...,,

......

×∈==

+++=+++

Example 2.33 revisited It turns out in fact that the following three

conditions are sufficient for a solution. 2m>N We now understand that we require to have

persistent excitation of the input signal sk. The Bezout condition must be satisfied for the

channels: no common zero! The solution itself we will provide later in the

context of SVD methods.

Markus Rupp

Resume The Hilbert space is a linear vector

space with the following three properties: 1) 2) 3)

Complete (Vollständig)MetricNormed (Normiert)There exists an inner product

Markus Rupp

Norms Definition 2.31: Let S be a vector space with

elements x. A real-valued function ||x|| is said to be a norm (length) of x, if the following four properties are satisfied:

Note that just as for metrics 0) follows from 1) and 3)! (can you show this property)?

)inequality (triangle )3

Cfor)2

0 ifonly and if 0)1

every for 0)0

∈≥

ααα

Markus Rupp

Norms Note, norms and metrics are related

concepts:

Example 2.49:

pppp xxdyxyxd =−= )0,(;),(

∞∞

[ ] )(sup)(

dttxtxL

∈∞∞ =−

Norms Why is l2 never smaller than l00?

With this method you can prove that

Markus Rupp

max 221max

1 ....ii

xx x x x xx ∞ ∞

= = = + ≥∑ ∑

n x x x∞ ∞

≥ ≥

Markus Rupp

Norms Definition 2.32: A normed, linear space is a

pair (S,||.||) in which S is a vector space and ||.|| is a norm on S.

Metrics of a normed linear space are defined by norms.

Definition 2.33: A vector is said to be normalized (Ger.: normiert, unit vector, Ger.: Einheitsvektor) if ||x||=1.

Except of the zero vector all vectors can be normalized.

Markus Rupp

Norms Note: The elements (vectors) of a linear

and normed space are not necessarily normalized vectors.

Only the space is normed, meaning that a norm exists for this space.

Norms Consider a sequence of vectors yk and a

limit y*. To evaluate whether yk converges to the

limit, we employ a norm:

Or, equivalently we can substitute xk=yk-y* and obtain:

Markus Rupp

0lim *

∞→

Markus Rupp

Norms Norm Equivalence Theorem 2.2: Let ||.|| and

||.||‘ be two norms on finite dimensional spaces Rn

(Cn,Qn,Zn,Nn,Bn), then we have:

Proof: It is to show that Without restricting generality we set (Ger.: oBdA)

||.||‘= ||.||2 and obtain for the lower bound:

0'lim,0lim == ∞→∞→ kkkk xx if only and if

xexexexx

=≤≤≤

∑∑ ∑

0,;' >≤≤ βαβα xxx

Markus Rupp

Norms For the upper bound β consider the case:

And we obtain:

With the above condition, x cannot be the zero vector and since ||.||>0 there must be a positive lower bound β:

Since the property is true for the l2 norm, it is also true for all other norms!

>≥= ββ

xxc β≤=2

'1'1' xxxxxxαβ

βα ≤≤→≤≤

Markus Rupp

Norms Note that the equivalence theorem is defined in terms of

Cauchy sequences. This is a consequence from the proof. Since if a particular norm tends to zero, then it does for every norm. The equivalence is to be understood in this sense.

Norms often find their application in terms of energy relations. It can be in form of average energy (l2-norm) or peak values (l∞-norm).

Thus, norms appear often when we describe systems. (Robustness, nonlinear systems, convergence of learning systems such as adaptive equalizers or controllers).

Markus Rupp

Norms Example 2.50: Hands-free telephone,

Ger.: Freisprechtelefon

+Far end speaker sk rk

Local speaker vk

2hh − :energyerror System

Markus Rupp

Norms Example 2.51: Consider an adaptive equalizer as

part of a receiver.

The equalizer has a cost function of the form

which it tries to minimize (adaptive process). The error at its output is given by an error vector:

h f+sk rkvk kk ss ≈ˆ

( )kk ssf −ˆ

],...,[]ˆ,...,ˆ,ˆ[ 11 kNkkkNkNkNkNkk eesssssse −+−+−−− =−−−=

Markus Rupp

Norms Example 2.51: Let the adaptive equalizer have

the property that the error signal from k-1 to k is mapped via the learning rule: y=g(x)=x3. Under which condition is the equalizer adaptive?

1......sup

...supsup?

−−−

−∈

−−

For |ek|<1 the error term at the output is always smallerthan at the input.

Markus Rupp

Norms Example 2.52: Consider the linear system in matrix-

vector-form.

Consider the ratio of input and output energy. When do we have a passive system, when an amplifier?

1 1 0 2

......

k N M k N M

N M kNN k

y h h xh h

y h xy h xy h h xy H x

− + − − − +

− − −

− −

)(sup?

kNlx Hf

Norms on Matrices In principle we can consider matrices

as vectors and apply the same norms, e.g., apply the l2 norm:

and obtain the Frobenius norm.

Markus Rupp

),min(

...)trace( 222

AAAA pH

++=== ∑∑= =

σσσ

Ferdinand Georg Frobenius (1849-1917)

Norms on Matrices Similarly

define norms on matrices

Markus Rupp

= =∑∑

Markus Rupp

Induced Norms Inner vector products are helpful for defining

norms. For example, the (quadratic) l2 norm can be written as inner vector product:

This is thus said to be an induced (Ger.: induzierte) norm. The norm is induced by the inner vector product.

Of the lp norms for vectors only the l2 norm is an induced norm!

xxxxxx n

Markus Rupp

Induced Norms Example 2.53: induced norm in L2[a,b]

Note that for real-valued signals we have:

How can this be formulated for complex-valued signals?

yxyxyx ,22

2=−−+

2)()(),()(

== ∫

dttxtxtxtx

Markus Rupp

Induced Norms Vector norms can also be used to induce norms on

multivariate functions. Consider the following example:

( )2,0

expsup

)exp(sup

),(sup

)exp(),(

2121],[

2121],[2

212121

xxxxxxf

normSelect

Markus Rupp

Induced Norms

Markus Rupp

Induced Vector Norms Matrix-norms are typically induced

vector-norms. Consider matrix A

Note that (spectral norm):

This is the largest singular value.

p 100ind,supsupsup =≠≠ =

( )AAAxAA Hx maxmax21ind,2

)(sup2

λσ === =

nmR ×∈

Induced Vector Norms

Markus Rupp

Spectral Norm Consider firstly a vector with M elements, based on the

sequence exp(jΩk):

In other words, all these vectors are linearly dependent , independent of their dimension M and frequency Ω.

If, however, M different frequencies are selected, M linearly independent vectors can be obtained.

[ ][ ]

[ ] kMT

xmjmMkjmkjmkjx

xjMkjkjkjx

Mkjkjkjx

)exp())1(exp()),...,1(exp()),(exp(

...)exp())2(exp(),...,exp()),1(exp(

))1(exp()),...,1(exp(),exp(

Ω=++−Ω+−Ω+Ω=

Ω=+−ΩΩ+Ω=

+−Ω−ΩΩ=

Markus Rupp

Spectral Norm Consider now the spectral norm for the vector xM,k:

0exp( ( ))

...exp( ( 1))exp( ( ))

(exp( )) exp( ) exp(

(exp( ))(exp( )) exp( ( 1))(exp( )) exp( ( ))

MM k l

h hj k l

j k l mj k l m n

H j j l

H jH j j l mH j j l m n

= Ω + + − Ω + + +

Ω Ω = = Ω Ω Ω + − Ω Ω + +

22,ind

, 2, 2,ind

(exp( ))exp( ( 1))exp( ( ))

sup sup (exp( ))M k

x M MM k l

H j xj l mj l m n

AxH j A

+Ω= Ω →∞

Ω = Ω Ω + − Ω + +

= → Ω =

Further Matrix Norms Recall:

Why is that? Consider an element yi of y=Ax

Markus Rupp

,ind,ind11

and:BUT

:generalin

max:maxrow

max:maxcolumn

∞∞

=≤≤∞

=≤≤

?maxmax1

∑∑

jjijxix

xAyxAy

We can maximize the expression if sign(xj)=sign(Aij) as then xjAij =|xjAij|

If we further restrict the elements of vector x, e.g., ||x||=1, then there are two possibilities

Markus Rupp

,ind,ind11and

∞∞== AAAA

[ ] ijj

maxmaxand10,...0,1,0,...,0,0

maxand11,...,1,1

1==→=

==→±±±=

∑∑∞

Markus Rupp

,ind,ind11and

∞∞== AAAA

maxmaxmax

maxmaxmaxmax

xAxAxxA

ixxxind

jjijxi

jjijixxxind

∑∑∑∑

∑∑

∑∑∑

=≤≤

≤≤=

=∞=∞

∞∞

Markus Rupp

Submultiplicative Property Some of the matrix norms (all p-norms) satisfy

the submultiplicative property:

Note that this property holds for arbitrary matrices, for example

But in this case the p-norms are different functions!

∞=≤ ,...2,1;ind,ind,ind,

pBAABppp

4332 , ×× ∈∈ RBRA

pppxByAxAB ≤

Submultiplicative Property Proof:

Markus Rupp

indpindpp

pxindp

pindpx

pxindp

supsup

Markus Rupp

Submultiplicative Property Not all matrix norms have this submultiplicative

property: Example 2.54:

∆∆∆

:have we For 1111

Submultiplicative Property A similar property, however, is true

for all unitarily invariant norms:

unitarily invariant norm: (C is unitary)

Markus Rupp

Matrix Norms Example 2.55: Consider a causal impulse response hk.

In which case is hk an allpass (amplifier)?

Obviously, the energy of the input sequence must be identical to the energy of the output sequence (for every sequence).

Note that the vectors xN+M-1,k and yN,k have different length!

1 1 0 2

......

k N M k N M

N M kNN k

y h h xh h

y h xy h xy h h xy H x

− + − − − +

− − −

− −

Markus Rupp

Matrix Norms We must thus have for an allpass:

This in turn requires that N∞. For the spectral norm we have at least:

supsup

limlimlim

−+≠

−+∞→−+∞→∞→

xindpN

pkMNNpkMNNNpkNN

least at havemust we

have we Since

( ) ( )indNN

jj HeHeH,2

limmax ∞→Ω

Markus Rupp

Matrix Norms An allpass is thus difficult to describe by

matrices of finite dimension or by the spectral norm.

The reason for the finite dimension is that the initial values need also to be considered to reflect the lossless property of the allpass. This property can be required over the entire observation horizon.

Markus Rupp

Matrix Norms Allpasses are better suited for state space forms. Note:

Example 2.56: yk=xk-2

00110001000,1

Markus Rupp

Matrix Norms Consider the

instantaneous energy:1

2 2 2 2 2 21 2 1 0 1 1

2 2 2 2 2 21 1

2 2 2 2 2 21 2 1 1 1

2 2 2 2 2 21 1

2 2 2 21 0

0 1 00 0 11 0 0

k k k k k k

N N N N N NN N

N k kk k

z zy x

z z y z z x

z y z x

+ + + + +

+ + = + +

+ = +∑ ∑

For all N

Markus Rupp

Matrix Norms Allpass as system with feedback

1+kzkz

Markus Rupp

Matrix Norms This more general form also allows for describing allpasses

by linear time-variant systems. Example 2.57: Let b=sqrt(1-a2); 0<a<1

.a-1bfor allpass an also describes

allpass. an describes

Markus Rupp

Matrix Norms Allpass

unitary! be to said are conditions these satisfythat , Matrices vectors arbitrary allfor

:havemust we allpass anfor this tocontrast In

have we systems,invariant -timelinear For

xindpN

supsup

−+≠

Matrix Norms Let us summarize:

The 2 induced matrix norm

tells us whether a system is passive or active

If a matrix is unitary, it describes an allpass.

Markus Rupp

)(sup2

λσ === =

A amplifier

A attenuator

Markus Rupp

Resume Norms are…

Metrics in a linear vector space. A norm defined over a linear vector space makes it to a

normed, linear vector space. The knowledge of one norm is in general sufficient due

to the Equivalence Theorem

An induced norm is… a norm, that is build by a „simpler construct“

(example: the inner vector product induces the p=2 norm)

0'lim,0lim == ∞→∞→ kkkk xx if only and if

Markus Rupp

Resume An induced matrix p-norm is…

a vector p-norm, applied to a matrix.

The spectral norm is… the vector p=2 norm induced to a matrix. In case of linearly time-invariant systems, it is identical with the

maximum magnitude of the transfer function.

Why is this true? Due to submultiplicative property of vector norms.

The submultiplicative property for matrices reads… .

pindppxAxA

BAAB ≤

Markus Rupp

Induced Vector Norms (Resume) Matrix-norms are typically induced

vector-norms. Consider matrix A

Note that (spectral norm):

This is the largest singular value.

p 100ind,supsupsup =≠≠ =

)(sup2

λσ === =

nmR ×∈

Markus Rupp

Resume Inner vector products can also be written in terms

of induced norms. In R:

yxyxyx ,22

2=−−+

yxyxyx

=−−+

Markus Rupp

Small Gain TheoremGeorge Zames(1934-1997) in 1966

Consider the following system with feedback.

Is it stable?

0,5q-1

Pole at-0,5

stable!

Markus Rupp

Small Gain Theorem Consider the following system with feedback.

Is it stable?

y=1/(1+h2)

0,5q-1

Markus Rupp

Small Gain Theorem The „Small Gain Theorem“ can be interpreted as

an extension from linear systems to nonlinear systems for the stability problem.

In linear time-invariant systems the gain of the open loop is responsible for stability (BIBO stability).

The Small Gain Theorem can be applied to non-linear as well as to time-variant systems.

Markus Rupp

Small Gain Theorem Consider for this the input output relation of a

system described by vectors: Definition 2.34: A mapping H is said to be lp-

stable if there exists two positive constants β,γ so that for all input sequences the following is true:

Definition 2.35: The smallest positive value γ, for which lp-stability is obtained, is said to be the gain of H (Ger.: Verstärkung).

Note: BIBO (Bounded Input Bounded Output) stability is equivalent to l∞- stability.

βγ +≤=pNpNNpN

NNNxHy =

Markus Rupp

Small Gain Theorem Consider now a system with feedback, comprising

of two systems G and H and corresponding gains γgand γh described by the following signals:

[ ][ ]

NNNNNN

yuGgGz

zxHhHy

Markus Rupp

Small Gain Theorem

gNzN uN

Markus Rupp

Small Gain Theorem Theorem 2.3: (Small Gain Theorem) If both gains

γg and γh are such that:

then the two signals gN and hN of the system with feedback are bounded by:

1<hgγγ

[ ]ghhNhNhg

hggNgNhg

βγβγγγ

+++−

Markus Rupp

Small Gain Theorem Proof: Start with:

the norm is given by:

The derivation for gN is analogue.

[ ]hggNgNhg

hggNgNNhg

ghNhNgN

βγβγγγ

ββγγ

+++−

++++≤

NNNNNN yugzxh +=−= ;

gNzN uN

Markus Rupp

Example 2.58 Let the automatic control

of a cell phone power amplifier look like this:

g2z2 u2=0

37,031

Markus Rupp

Example 2.58 Is this control stable? We describe G2 and H2 by their gains:

22,2222

37,037,0

Markus Rupp

Example 2.58

stable.-l thus ,0,372,5:gain loop open the for have thus We

37,09137,0

sup37.0

Markus Rupp

Example 2.60 Consider the following problem:

Let the shape of the signal s(t) be known at the receiver.

How is the filter g(t) to design so that the SNR at the output of g(t) is maximum?

matched filter (Ger.: Signalangepasstes Filter)Dwight O. North 1943

g(t)+s(t)

v(t)d(t)

Markus Rupp

Chauchy-Schwarz Inequality Theorem 2.4: In an inner-product space S in Cn

with induced norm ||.|| we have:

The inequality becomes an equality if and only if x=αy.

Proof: We start with the simple fact that

22, yxyx ≤

≥− yx α

Markus Rupp

Chauchy-Schwarz Inequality We find further that

The minimum is obtained for: Thus:

−+−=

+−=−≤

yyxxyx

ααα

yxxyx −=−≤ αα

22, yxyx ≤→

Markus Rupp

Chauchy-Schwarz Inequality We are not done!

For x=αy we obtain ||x-αy||=0. Due to the norm property , equality to zero

can only be obtained for || x-αy||=0 if the argument is zero.

Thus equality is obtained if and only if x=αy.

Markus Rupp

Chauchy-Schwarz Inequality The Cauchy-Schwarz inequality implies:

for S in Cn:

for the space of real-valued functions in the interval [a,b]:

( )( )yyxxyx HHH ≤2

∫∫∫ ≤

dttgdttfdttgtf )()()()( 22

Markus Rupp

Chauchy-Schwarz Inequality Example 2.59 Correlation coefficient

Remember: For the correlation coefficient we have:

Proof: Consider zero mean random variablesx‘=x-mx and y‘=y-my:

( )( )[ ]1

E111 ≤≤−⇒≤≤−

-y-xσσ

( )( )[ ]yx

-y-xσσ

[ ] ( )( )[ ]

[ ] [ ][ ] [ ]

xy22yx

y'x'y'x'y'x'

−−=

σσσσ

Markus Rupp

Chauchy-Schwarz Inequality Chauchy-Schwarz requires that

[ ] [ ] [ ][ ] [ ] [ ]

[ ][ ] [ ] 1

y'Ex'Ey'x'E

22y'x'

Markus Rupp

Still Example 2.59 Correlation Coefficient Note that E[.] defines an ensemble average. In practical problems this is often replaced by a

time average (ergodic processes). Cauchy Schwarz still holds then:

( )( )

( ) ( )∑∑

ykyxkx

Markus Rupp

Back to Example 2.60 Matched Filter (Ger.: Signalangepasstes Filter):

g(t)+s(t)

v(t)d(t)

∫∫∫∫

∞−

=−≤

ττττττττ

τττ

dgdrdtgdr

dtgrttd

dtgrtd

)()()()(

)()()(

Markus Rupp

Example 2.60 Matched filter:

g(t)+s(t)

v(t)d(t)

)()()()()(

)()()( 222

ταταττατ

ττττ

−≈−=⇔=−

≤ ∫∫∞

∞−

dgdrtd

:if only and if Identity,tpoint time suitable a to energyreceiver Maximize 0

Markus Rupp

Example 2.60 Graphical:

r(τ)g(t-τ)

∫∞

∞−

−= τττ dtgrtd )()()(

g(t-τ)

Markus Rupp

Alternative Example 2.60

g(t)+as(t)

v(t)d(t)

r(t)[ ] [ ]

)(),()(),(')'(

')'()'()()()(

)(),()()()()(

).()()(;1

ττσττσττσ

τττττδτστττ

ττττττττ

τδστ

ggtgtgdtg

dtgdtgdtg

stgdstgdstg

tt(t) s(t)

:zero is signal if energy Receiver

:zero is noise if energy ReceivervvEaE

with vanda signals random Consider

=−−=−=

−−−=

∫ ∫∫

∫∫

Markus Rupp

Alternative Example 2.60

g(t)+As(t)

v(t)d(t)

)()(2v

1)(),(

1)(),()(),()(),(

maxmax

:SNR Maximize

)(),(:only noise todueenergy Receiver

)(),()(

:only signal todueenergy Receiver

σσττ

ττσττ

ττσ

ταττ ==−

ssggstg

stgttS

stgstg

Markus Rupp

Example 2.61 A fixed symbol sequence sk in C with L symbols is

transmitted for the purpose of synchronization at the beginning of a TDMA frame.

Design an optimal synchronization filter fk. Received sequence without noise:

s Infodata ik s …[ ],...i,,i, 21 ssrT =

Markus Rupp

Example 2.61 Correlation filter:

Because of Chauchy-Schwarz we have:

kkkkkkf

ffrrfr

αα ≈=

= ∑−

,,,max2

Example 2.62Werner Karl Heisenberg (5.12.1901 – 1.2.1976)Hermann Klaus Hugo Weyl, (9.11.1885 – 8.12.1955)

Time-frequency uncertainty:

for F(jω) being the Fourier transform of f(t).

Markus Rupp

∫∞

∞−

∞−∞

∞−

ωωπ

ωωωπ

∫∞

∞−

∞−∞

∞−

ωωω

Example 2.62 Recall the following properties:

(partial integration with Dirichlet condition that f(t)0 for t+-oo)

Markus Rupp

∫∫∞

∞−

dttfdttfttf

djFdtfjt

)()()(2

)()()(

Peter Gustav Lejeune Dirichlet13. 2.1805 in Düren; † 5.5.1859 in Göttingen)

Example 2.62 Cauchy Schwarz says

Markus Rupp

1 ( )2

( ) ( ) ( ) ( )

( ) '( ) ( ) ' ( )

f t dt

f t g t dt f t dt g t dt

tf t f t dt t f t dt f t dt

−∞

∞ ∞ ∞

−∞ −∞ −∞

∞ ∞ ∞

−∞ −∞ −∞

= − ∫

∫ ∫ ∫

Example 2.62

Markus Rupp

∫∫≤

∫−

∞−

∞−∞

∞−

∞−∞

∞−

∫∫

ωωπ

ωωωπ

dttfdttftdttf

)(')()(21

Plancherel

Michel Plancherel (* 16.1.1885; † 4.3.1967 in Zürich) was a Suisse mathematician.

Example 2.62 Time-frequency uncertainty:

Q: for which functions do we obtain equality?

A: tf(t)=f‘(t)=exp(-αt2)

Markus Rupp

∫∞

∞−

∞−∞

∞−

ωωω

Alternative form (M.Vetterli et al. 2013) Define 1st and 2nd order moment:

Markus Rupp

Also works in the sequence domain:

Markus Rupp

Signal Processing 1 - nt.tuwien.ac.at · Resume Consider symmetric impulse response: h 0 =a, h 1...

Documents

Transcript of Signal Processing 1 - nt.tuwien.ac.at · Resume Consider symmetric impulse response: h 0 =a, h 1...

ENG 008 10 - deq.ufba.br · B = p C = 0 p A = − γ h p A d L H O h 2 = − γ Dividindo por H O 2 γ h A = − d L h (em coluna de H 2 O) Para maiores pressões efetivas negativas

Szerves kémiai szintézismódszerekkovacs/szint_7_jegyzet.pdf · H 3C H3C H H3C H CH 3C 3C 1° 3° (-)-α-pinén 3° 2° Cl izonorbornil-klorid H+ Cl-2 A karbokation generálása:

Testes para a média com variância conhecida ABORDAGEM CLÁSSICA –Estabelecer as hipóteses H 0 e H 1 –Definir o nível de significância α –Calcular a estatística.

Hersteller Maufacturer Fabricat Fabbricate SERION ELISA ... · PDF fileHersteller Maufacturer Fabricat Fabbricate Fabricate . 2 . 1 0 # . 1 2 " Výrobce Producent I j h b a \ h ^ b

Test de hipótesis · 1. H 0: μ= μ 0 vs. H A: μ ... cierta, a menos que la evidencia empírica demuestre lo contrario. •Si los resultados de la muestra indican que H 0 no se

MEINE SCHULE H A L L O ! I c h h e i β e L a u r i n e I c h b i n 1 2 J a h r e a l t. I c h w o h n e i n O s l y - C o u r t i l. I c h b i n i n d.

Maxima H Minus Reverse Transcriptase · Maxima H Minus Reverse Transcriptase Figure 1. Maxima H Minus Reverse Transcriptase generates full-length cDNA at a wider range of reaction

Organic Chemistry – chemistry of aromatics History · Organic Chemistry – chemistry of aromatics History ... - conjugation, resonance structures C 6 H 6 H H B A X H A X ... pyridine

5 A O G 8 2 ; C @ 8 M 0 A O A 8 ? ; 0 > 8 G 5 A 5 F 3 5 ...€¦ · 0 f ; : @ d g 2 ; c @ 8 m 0 a ; p a 8 ? ; 0 > p a : 5 f @ ; > p a g i g h 8 @ 1 h o a g h 8 a > j e f c l^k> z

NMR spectroscopy in characterization of organic solids ... · PDF fileNMR spectroscopy in characterization of organic solids - overview (1) SiH 3 H H H Si H3 H H H H H 180° 25° Si

1-Reacción con metales a) Na(s)+H 2 O = NaOH(ac)+½H 2 (g) agente oxidante H

Hõhere Analysis SS 06 Lõsungshinweis Aufgabe 1+3 Blatt 1 · Hõhere Analysis SS 06 Lõsungshinweis Aufgabe 4 Blatt 1 Sei (X;A;„) ein Ma…raum, und f: Y ! R. Zeigen Sie: a)

RMN 1H - RUA: Principal · 2016-04-25 · 562 Las configuraciones absolutas de los estereoisómeros del racémico A son: Cl C 6H 5 H H Me Br EtO _ 1 2 Cl H H C6H5 Br Me EtO _ 1 2

Consider the nuclear reaction: 11 Na 23 + α → [ 13 Al 27 ] → 12 Mg 26 + 1 H 1 Z X A + α → [ Z+2 C n A+4 ] → Z+1 Y A+3 + 1 H 1 Target Compound.

Harkiss, A. H., Bell, J. D. , Knuhtsen, A. , Jamieson, A ...eprints.gla.ac.uk/179501/7/179501.pdf · 1 Synthesis and Fluorescent Properties of -Pyridyl -Amino Acids Alexander H. Harkiss,

H. A. Tanaka

H 0 : H 1 : α = Decision Rule: If then do not reject H 0 , otherwise reject H 0 . Test Statistic:

T-16 PEMETAAN = z w 1 DAN HASIL PEMETAANNYA Oleh : H. A. ...

Bożena Żywicka , boshena@mif.pg.gda · t1 t2 t t2' 1’ T t h1 hx h(t) dla dźwięku 1 h(t) dla dźwięku 2 A h h2 h(t) dla kamienia 1 h(t) dla kamienia 2 h(t) dla lustra wody B

Organic Electronics: A Story of Science and Technology€¦ · Cambridge Society for the Application of Research 13 May 2013 . H H H H H H H H H H H H H H H H H H PPV: the prototypical