Signal Processing 1 - nt.tuwien.ac.at · Resume Consider symmetric impulse response: h 0 =a, h 1...

Signal Processing 1

Signal Spaces

Univ.-Prof.,Dr.-Ing. Markus RuppWS 18/19

Th 14:00-15:30EI3A, Fr 8:45-10:15EI4

LVA 389.166

Last change: 22.11.2018

2Univ.-Prof. Dr.-Ing.

Markus Rupp

Resume Consider the amplitude function of a filter:

A(Ω)=1+cos(Ω)+2cos(2Ω) Question 1: is this a symmetric filter? Answer: potentially yes, but only if

dφ(Ω)/dΩ =const. Question 2: What is the impulse response

if this is a causal symmetric filter? Answer: h0=1; h1=0,5; h2=1; h3=0,5; h4=1.

or further delayed versions of this


Markus Rupp

Resume Consider symmetric impulse response: h0=a, h1=1, h2=a. Which values of |a|<0,5 deliver a linear phase filter?

h0+h1exp(-jΩ)+h2exp(-2jΩ) =exp(-jΩ)[h0exp(jΩ)+h1+h2exp(-jΩ)]

=exp(-jΩ)[a exp(jΩ)+1+a exp(-jΩ)] =exp(-jΩ) [1+2a cos(Ω)]

=exp(jφ(Ω)) A(Ω) φ(Ω)=-Ω

What happens for |a|>0,5? A(Ω)=|1+2a cos(Ω)|Thus, for each value of a from R we have a

linear phase system but the phase function may jump.


Markus Rupp

Resume Consider symmetric impulse response: h0=a, h1=1, h2=-a. For which values of a is this a linear phase system?

h0+h1exp(-jΩ)+h2exp(-2jΩ) =exp(-jΩ)[h0exp(jΩ)+h1+h2exp(-jΩ)]

=exp(-jΩ)[a exp(jΩ)+1-a exp(-jΩ)] =exp(-jΩ) [1+2aj sin(Ω)]

Thus, for every value of a from C with Real(a)=0, this is a linear phase system. Since we have:

exp(-jΩ) [1+2aj sin(Ω)] =exp(jφ(Ω)) A(Ω) φ(Ω)=-Ω


Markus Rupp

Resume Question: Is the sampling of a continuous function

a linear operation? Answer: Yes

[ ] )2()1()2()1(

)1()1(

)()()]([

kk

k

ßfftßftfSftfS

+=+

=

αα


Markus Rupp

Resume Question: Is the interpolation of sampled values a

linear operation? Answer: Yes

(1) (2) (1) (2)

(1) (2)

(1) (2)

( ) ( )

( ) ( )

( ) ( )

( ) ( )

k kk

k k k kk k

k kk k

I f T f p t kT f t

I f ßf T f p t kT T ßf p t kT

T f p t kT ß T f p t kT

f t f t

α

α α

α

α β

∞

=−∞

∞ ∞

=−∞ =−∞

∞ ∞

=−∞ =−∞

= − =

+ = − + −

= − + −

= +

∑

∑ ∑

∑ ∑


Markus Rupp

Resume Question: Is the conversion from fk to f(t)

and vice versa unique under equidistant sampling?

Answer: Yes

What is the answer for equidistant sampling at arbitrary start point f(kT+To)?

)(

)()(

kTff

kTtpfTtf

k

kk

=

−= ∑∞

−∞=


Markus Rupp

Resume Note: Many different sequences fk

(i) ;i=1,2,… can lead to the same time-continuous function f(t). However, always a different interpolator is required.

Example:

)2/();(

)(

)()(

)2()1(

)2()2(

)1()1(

TkTffkTff

kTtpfT

kTtpfTtf

kk

kk

kk

+==

−=

−=

∑

∑∞

−∞=

∞

−∞=

Resume Given f(kT)=fk, how do we get

gk=g(kT)=f(kT+T/2)?


Markus Rupp

−+=

+= ∑

∞

−∞=

mTTkTT

fTkTfgm

mk 2sinc

2π


Markus Rupp

Learning goals Vector Spaces and Applications of Linear Algebra

in Signal Processing (6Units, Chapter 2) Metric spaces, sequences, Cauchy-sequences, supremum,

infimum, sparsity (Ch 2.1) Groups, Vector spaces, linear combination, linear

independence, basis and dimension, orthogonality,blind channel estimation (Ch 2.2, 2.7)

Norms and normed vector spaces (Ch 2.3-2.5) Applications of norms: robustness descriptions,

feedback systems with nonlinear elements: the small gain theorem

Inner vector products and inner product spaces, Hilbert and Banach spaces (Ch 2.8-2.9)

Induced norms, Cauchy-Schwarz inequality: matched filter and correlation coefficient, time-frequency uncertainty (Ch 2.6)

Sets, Spaces, and Vectors A set is a collection of distinct objects. A vector is an n-dimensional set of n ordered

elements. A space is a set with “some added structure”.


Markus Rupp


Markus Rupp

Example 2.1: Metric

Code bookCode word

1,0==⊂∈

BYBCx n

Decoder has to decide which one among the received y is allowed and most probable!

Presenter

Presentation Notes

Besser interpretieren


Markus Rupp

Example 2.1: Metric

Code Rate:

)(log2 Mk =# Info bits

# Code bits kn ≥

nM

nkR )(log2==

−),( kn Codexy

k2=


Markus Rupp

Example 2.1: Metric Decision of the „most probable symbol“

based on metric: Compare “distance” of received y to all possible,

allowed x. Symbol x with smallest (Hamming-) distance dH

is interpreted as correct.

( ) ∑ =⊕=

→×=n

i iiH

nnH

yxyxd

NBBdB

1

0

,

:,1,0


Markus Rupp

Example 2.2 We transfer two possible signal forms from two

different sensors: f(t) and g(t). The receiver has to find out whether the

distorted received signal form r(t) is closer to f(t) or to g(t).

We need a distance measure from r(t) to f(t) and from r(t) to g(t):

∫∫ −=−=b

a

b

a

dttgtrgrdvsdttftrfrd 22

22 )()(),(.)()(),(


Markus Rupp

Metric Spaces Consider a set X in R,C,N,Z,B… Definition 2.1: A metric

is a functional mapping to measure distances between objects/elements of the set X. In order to call such a distance a metric, the following properties need to be satisfied:

)(: 00 NRXXd +→×

Xzyxzydyxdzxdyxyxd

xydyxdyxd

∈+≤==

=≥

,, allfor );,(),(),()3 ifonly and if ,0),()2

),(),()10),()0

Metric Note that 0) follows from 1), 2), and

3) since:


Markus Rupp

0),( :thus3) and 1) todue

),(2),(),(),(0:have wethen

)2 todue 0),(

≥

=+≤=

=

yxd

yxdxydyxdxxd

xxd


Markus Rupp

Metric Spaces Example 2.3: Consider the following metric

d1(x,y)

defined over the vectors x,y. This metric is called l1 metric. (Manhattan distance)

Example 2.4: Consider the following metric dp(x,y)

( ) ∑ =

+

−=

→×n

i ii

nn

yxyxd

RRRd

11

01

,

:

( ) ( ) pn

i

piip

nnp

yxyxd

RRRd/1

1

0

,

:

∑ =

+

−=

→×


Markus Rupp

Metric Spaces From the lp metrics the l1 and l2 metric as well as

the l∞ metric are of particular interest:

Example 2.5: Hamming Distance

( ) ( )iini

pn

i

piip

nn

yx

yxyxd

RRRd

−=

−=

→×

≤≤

=∞→∞

∞

∑1

/1

1

max

lim,

:

( ) ∑ =⊕=

→×=n

i iiH

nnH

yxyxd

NBBdB

1

0

,

:,1,0


Markus Rupp

Metric Spaces Definition 2.2: A Metric Space (X,d) consists of

a set X and a metric d, valid on this set.

Example 2.6: The set Rn (vectors with n entries from R) together with the metric d2(x,y) builds a metric space.


Markus Rupp

Metric Spaces Note, a metric defines the distance to the zero

vector for p>=1:

A metric thus allows for statements about sizes (lengths,areas, volumes, etc for p=2) of objects. Tied to this is the existence of such objects.

Example 2.7: Consider the infinite long series of real or complex valued numbers xi, i=0,1,...,oo, with the property

∑∞

=

∞<0i

pix

( ) ( ) ( ) pn

i

pi

pn

i

pip xxxd

/1

1

/1

100, ∑∑ ==

=−=

( ) ( ) ∞<= xdxd pp 0,


Markus Rupp

Metric Spaces Together with an lp metric, such sequences form a

metric space called lp(0,∞) space (Ger.: Folgenraum).

Consider on the other hand the sequence xi, i= -∞,...,-1, 0,1,...,∞ with the same properties and same metric, we have the lp(-∞,∞) space.

Consider the metric p ∞, for sequences for which: |xi| is bounded for every i, i.e., |xi|<M, we obtain the l∞(0,∞) or l∞(-∞,∞) space, respectively.

nnnnn yxyxd −=∞ sup),(


Markus Rupp

Supremum and Infimum Definition 2.3. (Supremum): Consider the set S in

R,Q,Z,N… with the elements xi. The smallest number z in R, for which we have:

is called supremum (sup) of the set S. It is the least upper bound.

If there is no number in R that is larger than the largest element in S, then we have sup(S)= ∞.

ixz i allfor ;≥


Markus Rupp

Supremum and Infimum Definition 2.4 (Infimum): Consider the set S in R

with the elements xi. The largest number z in R, for which we have:

is called Infimum (inf) of the set S. It is also called the greatest lower bound.

If there is no more number in R smaller than the smallest element in S, then we have inf(S)=- ∞.

ixz i allfor ;≤


Markus Rupp

Supremum and Infimum Example 2.8: Let S=(3,6) be an open set,

then: inf(S)=3, sup(S)=6

Let T=[4,8), then: inf(T)=4, sup(T)=8 Let U=[2,∞), then: inf(U)=2, sup(U)=∞

Why do we not simply select the maximum (minimum) of S?

Presenter

Presentation Notes

Betrachte 1-1/n! was ist das maximum davon, wenn n=1,2,…sup()=1,inf()=0


Markus Rupp

Metric Spaces Metric spaces can also be formed with particular

properties:

Example 2.9: Let lh(0,∞) be a metric space of sequences in which sequences exist whose quadratic sum is finite (finite energy sequence). Thus, xi in lh(0,∞)=lp=2(0,∞) means, that

∞<∑∞

=0

2

iix


Markus Rupp

Metric Spaces Metric spaces can also be defined over

functions rather than a set of numbers (note that functions are in a way sets of numbers).

These are called metric spaces over functions (Ger.: Funktionenraum).


Markus Rupp

Metric Spaces Definition 2.5 (p-metric): Let X be a set of real-

or complex-valued functions, defined on the interval [a,b] with b>a, with p>=1, that have the property:

The metric over the functions x(t),y(t) from X is then:

(Lebesgue Integral)

∞<≤

−= ∫ pdttytxyxd

pb

a

pp 1;)()(),(

/1

∞<≤∞<∫ pdttxb

a

p 1;)(


Markus Rupp

Metric Spaces The space with the metric dp over the functions is

called Lp space.

For p∞ we have for bounded functions (from above and below) :

For more details on Lp, read Ch 2.1.3

btatytxyxd

txbat

≤≤−=

∞<

∞

∈

;)()(sup),(

)(sup ],[


Markus Rupp

t

0 1 2 3 4 5 6 7 8 9 10

x(t)

0

1

2

3

4

t

0 1 2 3 4 5 6 7 8 9 10

x(t)

0

1

2

3

4

ε

ε

−

+

)()(),(

)(

txtxtx

tx

o

mo

o

)(

)()(

2

1

tx

txtx

oεε

εε

>>>><<

∞

∞

),(),(),(),(

2

1

22

12

xxdxxdxxdxxd

o

o

o

o

εε

<<

∞ ),(),(2

mo

mo

xxdxxd

Metric: Problems with Pytagoras?


Markus Rupp

Under which dp metric is the picture correct?

Presenter

Presentation Notes

3^p+3^p=5^pP=log2/(log(5/3))=1.36

p-Metrics: consider iso-metricἴ iσος=equal


Markus Rupp

p=1

p=2p=4 p=100

p=1/2

p=1/4

inflation

deflation( )

pp

p

xx

xdxx

x

21

2

1

1

+=

=

=

Presenter

Presentation Notes

Iso-terme, iso-bare

Metrics for 0<p<1 What happens if we select 0<p<1? In this case

does not satisfy the triangle equality (subadditive property)


Markus Rupp

( ) ( ) pn

i

piip yxyxd

/1

1, ∑ =

−=

Metrics for 0<p<1 Example 2.10: select the three

twodimensional points x=(1,0),y=(0,1) and z=(0.5,0.3)

compute


Markus Rupp

2122

1

2 21 12 22 2

1 1

1.57 2.38

4

3.96

ni ii

n ni i i ii i

x y

x z z y

=

=

= =

= =

− =

> − + − =

∑

∑ ∑

Metrics for 0<p<1 Definition 2.6: For 0<p<1, the

following expression is a metric:

Example 2.10 again:


Markus Rupp

( ) ( )∑ =−=

n

i

piip yxyxd

1,

( )122

1 12

1 12 22 2

1 1

1.25 1.54

, 2

2.798

ni iip

n ni i i ii i

d x y x y

x z z y

=

==

= =

= =

= − =

< − + − =

∑

∑ ∑

d0-Metric Is there also a d0 metric? Answer: yes but the definition

causes some mathematical difficulties as discontinuities occur for


Markus Rupp

( ) ( )∑ =→ −=n

i

piip yxyxd

100 lim,

=

=elsex

x;1

0;00

Sparseness Definition 2.7: In practise this definition

of d0 is often given in the form of

d0(x)=Σxi0

which is truly a counter for sparseness. But note, this is not a metric (norm)

sometimes it is called a pseudonorm


Markus Rupp

Example 2.11Reconsider Sampling For bandlimited signals we found:

Given the interpolation function, what is the optimal sequence gk to minimize

Answer: gk=fk=f(kT)min=038

Univ.-Prof. Dr.-Ing. Markus Rupp

−

−=

∑

∑

∞

−∞=

∞

−∞=

)(sinc),(min

)(sinc)(

2 kTtT

gtfd

kTtT

ftf

kkg

kk

k

π

π

2/12

)(sinc)(min

−−= ∫ ∑

∞

∞−

∞

−∞=

dtkTtT

gtfk

kgk

π

Example 2.11Reconsider Sampling Given a set of interpolation functions pm(t) :

Find the pair (gk,pm(t));m=1,2,..,M that results in a sparse representation of f(t)!

basic concept for modern speech coding (CELP)


Markus Rupp

( )

TktkTtpgtf

pgd

m

k

kk

Mmmkgk

00

,...,2,10

0);()(

|min

0

≤≤−= ∑=

=

that such

Presenter

Presentation Notes

Code excited linear prediction

CELPCodebook excited linear prediction We can alternatively also select from a set of

excitation sequences g1,k,g2,k,…,gM,k. The interpolator can be constructed in form of a

fixed part with a linear filter (predictor) before:

The task now is to find the following minimum:


Markus Rupp

−= ∑

−

=

)(sinc)(1

0kTt

Tptp

P

kk

π

−+ ∑

=

0

,0

,2 )(),(minmink

kkmDpg kTtpgttfd

lkm

prediction for tD>0

CELPCodebook excited linear prediction

Problem is split into a search over a fixed set (gm,k) and the minimization of a d2-metricLeast Squares (Chapter 3)


Markus Rupp

21

)(

0

2

0

1

0,

21

)(

0

2

0,

0,2

0 0

,

0 0

,

0

,

)(sinc)(minmin

)()(minmin

)(),(minmin

−−−+=

−−+=

−+

∫ ∑ ∑

∫ ∑

∑

+

=

−

=

+

=

=

TkP k

k

P

l lkmDpg

TkP k

kkmDpg

k

kkmDpg

dtkTlTtT

pgttf

dtkTtpgttf

kTtpgttfd

lkm

lkm

lkm

π


Markus Rupp

Metric Spaces lp(0,∞) : Space of causal sequences lh(0,∞) : Space of causal sequences with

finite energy (p=2) lp(-∞,∞): Space of non-causal sequences

(this includes lp(0,∞)) Lp(0,∞): Space of causal functions Lp(-∞,∞): Space of non-causal functions (C[a,b],dp): Space of continuous functions


Markus Rupp

Metric Spaces Example 2.12 Filter design: a linear time-discrete filter with linear

phase H(exp(jΩ)) =a0+a1exp(jΩ)+a1exp(-jΩ)=a0+2a1cos(Ω) is to design, so that it follows a desired amplitude function |Hd(exp(jΩ))|=1 for |Ω|< ΩG (zero else) most optimally.

( ) ( )

( )

[ ] [ ]

)(;)sin(;

)sin(42)1(min

)cos(2021)cos(21

21min

)cos(221min

21min

10

121

20

20

20,

210

210,

2

10,

2

10

10

10

series Fourier cmp.

Rest

ππ

ππ

ππ

π

ππ

π

π

π

GG

GG

aa

aa

jdaa

jjdH

aa

aaaaa

daadaa

daaeH

deHeH

G

G

Ω=

Ω=

Ω−++−−Ω

=

ΩΩ−−+ΩΩ−−=

ΩΩ−−=

Ω−

∫∫

∫

∫

Ω

Ω−

−

Ω

−

ΩΩ


Markus Rupp

Minimization ofquadratic metric d2

ΩG

Example 2.13. Recall Paley Wiener Given an amplitude function Ad(Ω), what is

a valid filter function F(ejΩ), i.e.Ad

2(Ω) =|F(ejΩ)|2

Different to the previous example the full information of the desired Filter is not known, only the amplitude function is given.

We recall that linear phase filters are well described by their amplitude function.


Markus Rupp

Example 2.13. Recall Paley Wiener Linear phase filters have symmetric

impulse responses


Markus Rupp

Example 2.13. Recall Paley Wiener Problem formulation:

Recall that:


Markus Rupp

Example 2.13. Recall Paley Wiener For which we find:

For l=0,1,…,N. From there we can compute the original coefficients fn.

Thus, given Ad, we now know how to compute F.


Markus Rupp

Resume What is a set? What is a vector? What is a space? What is a metric space?


Markus Rupp


Markus Rupp

Resume Definition: Space X Objects (numbers,

sequences, functions…) with structure Definition: Metric space (X,d) space X on

a metric d is defined as

Xzyxzydyxdzxdyxyxd

xydyxdyxd

∈+≤==

=≥

,, allfor );,(),(),()3 ifonly and if ,0),()2

),(),()10),()0

Resume When do sequences in metric spaces

with lp metric exist?


Markus Rupp

∑∞

=

∞<0i

pix


Markus Rupp

Resume lp(0,∞) :

Metric space of causal sequences with p metric

lh(-∞,∞): Metric space of non-causal sequences with 2 metric


Markus Rupp

Metric Spaces Robustness (Ger.: Robustheit):

Often it is important not to optimize the mean value (e.g., mean data rate, BER) but the value at worst case condition.

If a systems shows to be insensitive to the worst cases, it is called robust.


Markus Rupp

Metric Spaces Robustness is often measured in terms of energy Let xi be the input sequence and yi the output sequence of a

system T. The output is assumed to be distorted by an unknown sequence vi. The form of distortion is not necessarily additive or even known. Furthermore, the M initial states zi of this system are also not known. Assume that there exists a reference system without distortion having an output signal yi

(R). The influence of such distortion can be described be the

following expression:( )

2

0

2

1

2

0

2)(

γ≤+

−

∑∑

∑

==

=N

ii

M

ii

N

ii

Ri

vz

yy


Markus Rupp

Metric Spaces Or somewhat more precise:

Assume now that there exists several possibilities for the realization of this system T(F) depending on a specific strategy. Then the robustness criterion reads:

( )2

0

2

1

2

0

2)(

),0(),,1(sup γ≤

+

−

∑∑

∑

==

=

∈∈N

ii

M

ii

N

ii

Ri

NlvMlz vz

yy

hh

( )

∑∑

∑

==

=

∈∈ +

−

N

ii

M

ii

N

ii

Ri

NlvMlzFvz

yy

hh

0

2

1

2

0

2)(

),0(),,1(supinf


Markus Rupp

Example 2.19: Robustness A power amplifier PA in a cell phone (Ger.: Handy)

can be described by the following nonlinear mapping:

The variable ρ= ρi(T) describes the influence of temperature, aging and more.

In order to define the robustness of the output signal with respect to ρ, the following expression is considered:

iiii

iii vx

xx

y ++

= 2

2

1 ρ

ρ

( )2

0

2

0

2

0

2)(

),0(,sup γ

ρρ≤

+

−

∑∑

∑

==

=

∈N

ii

N

ii

N

ii

Ri

Nlv v

yy

h

PA yiT


Markus Rupp

Example 2.19: Robustness If it is possible to reduce the influence of

ρ by new technologies or improved circuitry, a smaller factor γ is the result:

The improvement is achieved by higher robustness (smaller γ).

Note: Robustness is not sensitivity.

( )22

0

2

0

2,

0

2)(

),0(,sup γγ

ρρ<≤

+

−

∑∑

∑

==

=

∈newN

ii

N

inewi

N

ii

Ri

Nlv v

yy

h

PA yiT

Contr.

+


Markus Rupp

Metric Spaces Convergence of sequences

Definition 2.8 (limit): If there exists to every distance δ a number no so that d(xn,y)< δ for each n>no at fixed value y, the sequence xn is said to be convergent to y.

y is called the limit (Ger.: Grenzwert) of xn. All points in an arbitrarily small distance to y are called the

neighborhood (Ger.: Nachbarschaft) of y.

nn

n

xyyx

∞→=→lim


Markus Rupp

Metric Spaces Example 2.14: the following two sequences

diverge:

Definition 2.9: If a sequence xn returns infinitely often to a neighborhood of z, then we call this point z a limit point (Ger.: Häufungspunkt, Grenzpunkt). E.g., the sequence bn takes on the limit points z=0 and z=2.

If limit points exist, then there must be subsequences (partial sequences, Ger.: Teilfolgen) xn that converge.

nn

n

bna

)1(1

2

−+=

=


Markus Rupp

Metric Spaces The largest limit point of a sequence xn is called

limes superior, or

The smallest limit point of a sequence xn is called limes inferior, or

A sequence converges, if:

nn x∞→suplim

nn x∞→inflim

nnnn xx ∞→∞→ = supliminflim


Markus Rupp

Metric Spaces Example 2.15: Consider the sequence

There are two limit points. The subsequence c0,c2,c4,.. takes on the limit 4, while the subsequence c1,c3,c5,.. takes on the limit 0.

12 2 ( 1) ; 1,2,3,...

limsup 4 but : sup 4,5liminf 0

nn

n n n n

n n

c nn

c cc

→∞

→∞

= + + ⋅ − =

= ==


Markus Rupp

Metric Spaces Definition 2.10: A sequence xn in a metric

space (X,d) is called Cauchy-sequence, if there exists for every ε>0 a N(ε)>0 so that d(xn,xm)<ε for every m,n>N.

It is possible to prove that sequences that converge are Cauchy-sequences. The opposite is not always true.

Presenter

Presentation Notes

Problem der vorherigen Definition ist ja dass man den grenzwert kennen muss


Markus Rupp

Metric Spaces Example 2.16: Let X=C[-1,1] be a set of

continuous functions and fn(t) a sequence of functions, defined by

Consider the metric space (X,d2) with

>≤≤−+

−<=

ntntnnt

nttfn

/11/1/12/12/

/10)(

( )∫−

−=1

1

22 )()(),( dttgtfgfd


Markus Rupp

Metric Spaces

nn /1/1

1

− t

)(tfn

Example 2.16:


Markus Rupp

Metric Spaces We find

With d20 for large m and n, m-n=k<oo. We thus conclude that it is a Cauchy sequence.

Note, however, the function in the limit is:

This function is not continuous and thus not in X=C[-1,1]. The sequence is thus not convergent in X!

<−

>−

=nm

mnm

nmn

nm

tftfd mn

;6

)(

;6

)(

))(),((

3

2

3

2

2

>=<

=0102/100

)(ttt

tf


Markus Rupp

Metric Spaces The reason why the Cauchy-sequence does

not converge in X, can be interpreted as a hole in the set of functions X.

If we had defined X as the set that included discontinuous functions as well, then we had obtained a Cauchy sequence that converges.

Definition 2.11: A metric space (X,d) is called complete (Ger.: vollständig), if every Cauchy-sequence converges in X.


Markus Rupp

Metric Spaces Gibbs‘ Phenomenon You are probably well aware of the fact that

periodic functions can be described by Fourier series.

Example 2.17: Consider the following periodic signal of period 1 in L[-0,5 , 0,5]:

with the Fourier-series:

<<≤≤−−<≤−

=5,025,0025,025,0125,05,00

)(tt

ttf

( )

−−

−+= ∑

=

+

tkk

tfn

k

k

n ππ

)12(2cos12

)1(25,0)(1

1

Josiah Willard Gibbs (11.2.1839 –28.4.1903) was an American scientistin Physics, Chemistry, Mathematics


Markus Rupp


Markus Rupp

Metric Spaces Let‘s run this to noo Note that in this case we have:

0))(),((lim0))(),((lim 2

>=

∞∞→

∞→

tftfdtftfd

nn

nn

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5-0.2

0

0.2

0.4

0.6

0.8

1

1.2

)(500 tfn=

Example 2.18 Let us consider the problem of finding a

square root of a number A (e.g. A=3 in N) We thus want to find a solution for x2-A=0 An iterative algorithm (Heron) can be

derived (Newton-style):


Markus Rupp

21n

n

nxAx

x+

=+

Heron from Alexandria (Mechanicus; † > 62) Greek mathematician and engineerMethod is known since 1700 BC.

Presenter

Presentation Notes

Babylonian root calculation

https://de.wikipedia.org/wiki/62

Example 2.18 Obviously, starting with a value from

Q, results in a new value from Q:

For noo, we obtain a value from R!


Markus Rupp

QabA

ba

xbax

xAx

x

nn

nn

n

∈+

=→=

+=

+

+

2

2

1

1

Rules in the Vector Space Obviously we want to „do“ something

with the objects in our set(space) We can stretch them

We can add them


Markus Rupp

x xα→

,x y x y→ +

Binary operation Definition 2.12: A binary operation *

on a set S is a rule that assigns to each ordered pair (a,b) of elements from S some element from S. Since the operation ends with an element in the same set, we call this also a closed operation.

Example 2.20: S=Z, *: a*b=min(a,b)


Markus Rupp


Markus Rupp

Vector Spaces Definition 2.13: A linear vector space S over a set of

scalars T (C,R,Q,Z,N,B) is a set of (objects) vectors together with a binary additive „+“ and a scalar multiplicative „.“ operation, satisfying the following properties:

1) S is a group under addition. 2)

3) W.r.t. the multiplicative operation there exists an Identity (One) and a Zero element:

yaxayxaxbxaxbaxabxbaSxa

SyxTba

+=++=+=∈

∈

)()(,)()(,

:,, have we in and everyFor

00;1 =•=• xxxIdentity elementw.r.t „+“!


Markus Rupp

Groups Definition 2.14 (Group): A set S for which a binary

operation * (operation w.r.t two elements of S) is defined, is called a group if the following holds: 1) for each a,b in S it holds that: (a*b) in S 2) there exists an identity element e in S, so that for every

element a in S: e*a=a*e=a. 3) for each element a in S there exists an inverse element b in

S, so that: a*b=b*a=e. 4) The binary operation * is associative, i.e.,: (a*b)*c=a*(b*c)

The group is denominated by (S, *). If, furthermore, for each pair a,b in S it holds that a*b=b*a

(commutativity), then the group is called commutative or Abelian (Ger.: Abelsch).


Markus Rupp

Groups Example 2.21: Let S be the set of vectors of a

particular dimension. S forms a group w.r.t the additive operator, if the following properties are satisfied:

( ) ( )zyxzyxSzyx

xyyx

SySxxxx

S

SyxSyx

++=++∈

−==+

∈∈=+=+

∈+∈

:that holdsit eachFor :eassociativ is operation additive The

that so , second a exists there element eachFor

that so , in element identity an exists There

:that holdsit everyFor

,,)4

;0

)300

0)2

,)1

Presenter

Presentation Notes

Note 3) geht bei * statt + auch , aber nicht für das element 0!


Markus Rupp

Rings Definition 2.15 (Ring): A set S for which two

binary operations + and * are defined, is called a ring if the following holds: (S,+) is a commutative group (Abelian) The operation * is associative. Distributivity holds w.r.t. + for all scalars:

a(b+c)=ab+ac, (a+b)c=ac+bc. A ring is denominated by (S,+, *). Note: for * there does not need to be an identity

or inverse element. If there exists additionally the inverse element to

*, then it is called skew field or division ring (Ger.: Schiefkörper).


Markus Rupp

Fields Definition 2.16 (Field): A set S equipped

with two binary operations + and * is called a field (Ger.: Körper), if: 1) (S,+) is an Abelian group 2) (S\0,*) is an Abelian group 3) The operations + and * distribute.

If the set is finite, i.e., |S|<oo, we talk about finite groups, finite rings, finite fields….

Presenter

Presentation Notes

\0 wegen a*(1/a)=1 geht nicht für null! Aber sonst gibt es inverses Element der *


Markus Rupp

Fields Example 2.22: The Galois Field GF(2) is a

field. The elements of the field are 0 and 1. Define the operations + and *

a b a+b0 0 00 1 11 0 11 1 0

a b a*b0 0 00 1 01 0 01 1 1

Presenter

Presentation Notes

Not a skew field due to *


Markus Rupp

Fields W.r.t. + S needs to be an Abelian Group:

a+b is in S. Identity element is 0: 0+0=0, 1+0=1. Inverse element exists: 0+0=0,1+1=0. Associativity: (a+b)+c=a+(b+c). Check the remaining properties yourself...


Markus Rupp

Fields Example 2.23:

(R,+,*), (Q,+,*),(C,+,*) are fields; (N,+,*),(Z,+,*) not!

Presenter

Presentation Notes

Inverse element does not exist in N for + and *, not in Z for *


Markus Rupp

Group

Relation to each other

Ring AbelianGroup

Vector space

SkewfieldField


Markus Rupp

Vector Spaces Example 2.24: for finite dimensional linear vector

spaces: 1) Consider the linear vector space in R4 (set of

quadruples, Ger.: Menge der Quadrupel)

2) The set of m X n matrices with real-valued elements.

3) The set of polynomials of degree 0..n with real-valued coefficients.

=+

=+

−

=

=

2121913

23;

0476

;

2025

;

2451

yxyxyx


Markus Rupp

Vector Spaces Still Example 2.24: Consider the (3,2)

single parity check code in GF(2)3 with the elements:V=[000],[011],[101],[110]

„+“ now means „exclusive or“

Do the remaining elements W=[111],[100],[010],[001] also form a linear vector space?

Presenter

Presentation Notes

Antwort: nein, da 111+100=011 aus der anderen menge ist!


Markus Rupp

Vector Spaces Example 2.25: for infinite-dimensional linear

vector spaces:

1) Consider the set of infinitely long sequences

xn

2) The set of continuous functions over the

interval [a,b] C[a,b]

3) The set of functions in LpLp[a,b]


Markus Rupp

Vector Spaces Definition 2.17: Let S be a vector space. If V is a

subset of S such that V itself is a vector space then V is called a subspace (Ger.: Unterraum) of S.

Example 2.26: Let S be the set of polynomials of arbitrary degree (>6) and V the set of polynomials of degree less than 6. Then V is a subspace of S.

Example 2.27: A (n,k) binary linear block code is a k-dimensional subspace of GF(2)n.

Presenter

Presentation Notes

Dim=kardinalität der hamel basis, es gibt k lu vektoren Grassmannian Manifold: subspaces with structure e.g. lines through origin in 3D


Markus Rupp

Vector Spaces Definition 2.18: Let S be a linear vector space

over R and T a subset of S. A point x of S is called linear combination (Ger.: Linearkombination) of points in T, if there is a set of points pi ;i=1,2,...,m in T and a finite set of scalars ci ;i=1,2,...,m in R, so that:

Note that the set T does not need to be finite.

mm pcpcpcx +++= ...2211

Presenter

Presentation Notes

All subsets of cardinality m need to be linearly independent-> can be tough to proof!


Markus Rupp

Vector Spaces Example 2.28: Let S=C(R), the set of continuous

functions over the complex (real) numbers. Let, furthermore, p1(t)=1; p2(t)=t, p3(t)=t2. A linear combination of such functions is given by:

Consider the polynomial x(t)=-1+5t+t2, and the function p4(t)=t2-1.

Obviously, the description is not unique. The number of required coefficients varies.

2321)( tctcctx ++=

)()(5)(2)()(5)(

)()(5)()(

42

4321

321

tptptptptptp

tptptptx

+=+−+=

++−=


Markus Rupp

Vector Spaces Definition 2.19: Let S be a linear vector space

and T a subset of S. The subset T (pi ;i=1,2,…,m) is said to be linearly independent (Ger.: linear unabhängig), if for each nonempty linear subset T the only finite set of scalars satisfying the equation

is the trivial solution c1= c2 =...= cm =0.

If the above equation is satisfied by a set of scalars that are not all equal to zero, then the subset (pi ;i=1,2,...,m) is called linearly dependent(Ger.: linear abhängig).

0...2211 =+++

mm pcpcpc


Markus Rupp

Vector Spaces Example 2.29: The previously presented

polynomials p1(t)=1; p2(t)=t, p3(t)=t2, p4(t)=t2-1 are linearly dependent, since:

The polynomials p1(t),..., p3(t) are linearly independent!

The vectors p1=[2,-3,4],p2=[-1,6,-2] and p3=[1,6,2] are linearly dependent since:

0)()()( 431 =+− tptptp

0354321

=++ ppp


Markus Rupp

Vector Spaces Example 2.30: Consider the complex numbers

z=r+j i in R2 in vector form:

This is a set describing z and z*. Note that both elements are linearly independent as long as i is unequal to zero!

−

=

ir

ir

T ,


Markus Rupp

Vector Spaces Example 2.31: Consider band-limited functions or

sequences xk (also band-limited random processes) that only exist in a frequency range S. For a sequence fk (linear time-invariant system) existing in the complementary space of S we find:

Band-limited signals are said to be linearly dependent! Note however, that the property “finite number of

elements” is not necessarily satisfied.

( ) ( )( ) ( ) ( )

knk

kknk

kn

jjjkn

kkn

jjn

ff

ff

eXeFeYf

SeFSeX

−

−

−∞=−

∞

=

ΩΩΩ−

∞

−∞=

ΩΩ

∑∑

∑

−−=

==⇔==

∈Ω=∉Ω=

xxx

xy

for for with xLet

1

010

11

00

0;0


Markus Rupp

Example 2.33 Consider blind channel estimation (for

equalization):

h1

h2

sk

g1

g2

sk(1)

sk(2)

-0

rk(1)

rk(2)


Markus Rupp

Example 2.33 Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

(1)]T

We obtain the equation:

Which properties must the receive vectors rk(1) and rk

(2)

have, so that the problem can be solved uniquely? Vectors rk

(1,2) need to be linearly dependent!

[ ] 0,...,,,,...,,

......

2

21

1

11

)2()2(2

)2(1

)1()1(2

)1(1

)2(2

)2(222

)2(121

)1(1

)1(212

)1(111

==

−

−

+++=+++

gR

g

gg

g

rrrrrr

rgrgrgrgrgrg

m

mmm

mmmm

Presenter

Presentation Notes

Diese müssen linear abhängig sein, sonst ist g=0


Markus Rupp

Vector Spaces At this point several interesting questions arise:

Under which conditions is the linear combination of vectors unique?

Which is the smallest set of vectors required to describe every vector in S by a linear combination?

If a vector x can be described by a linear combination of pi ;i=1..m, how do we get the linear weights ci ;i=1..m?

Of which form do the vectors pi ;i=1..m need to be in order to reach every point x in S?

If x cannot be described exactly by a linear combination of pi ;i=1..m, how can it be approximated in the best way (smallest error)?


Markus Rupp

Resume Space Metric complete

Convergence Cauchy-Sequence

Set vs. Vector


Markus Rupp

Resume Consider the following sequences in the

metric space l∞(1,∞):

Consider a classical Gaussian noise sequence from N(0,σ2). Is such a sequence in the metric space lp(-∞,∞)?

1),(

1;

11

≥−>

+=

+=

∞ nyxyxd

nny

nx

nnnn

nn

allfor ;that Proof

Presenter

Presentation Notes

d_infty is supremum, thus one , obtained for ninfty!


Markus Rupp

Resume What can be said about the convergence of

these causal sequences (n=1,2,..)?

Which of these sequences is in the metric space l2(1,∞), which in l∞(1,∞)?

[ ]

[ ]

n

n

n

n

nn

nn

nd

nc

nb

na

235,0

11

)1(111000

11

)1(111

+=

+=

−−

++=

−−+=

Presenter

Presentation Notes

l2 heisst, dass sum x_n^2 <M !!! Also nicht a,b,c; loo: alle


Markus Rupp

Resume Consider the following sequence in Q in the

metric space (Q,d∞):

? in sequence-Cauchya thisIs

converge? sequence thisDoes

,...100000141421,

1000014142,

10001414,

100141,

1014,

11

Q

Not in Q

Yes!

Resume Problems in Q: Consider Is this convergent in Q? Similarly, consider


Markus Rupp

21

1n

sm

nm ∑

=

=

!1

0 ns

m

nm ∑

=

= en

m

nm =∑

=∞→ !

1lim0

61lim

2

21

π=∑

=∞→ n

m

nm


Markus Rupp

Resume Which of the following sets of vectors are LI and which not?

−

12363

,

9642

,

4321

137

,1

5,

53

,21

2000

,

3300

,

4220

,

5111

4321

,

0321

,

0021

,

0001


Markus Rupp

Resume Given the set pi and a vector x, how do we obtain the linear

weights?1 21 2

... m mp p p xα α α+ + + =

( )

1

1 2

11

1 2

1

1 2 1 2 1 2

, ,...,

, ,...,

, ,..., , ,..., , ,...,

m

m

m

m

H H

m m m

p p p x

p p p x

p p p p p p p p p x

α

α

α

α

−

−

= =

=


Markus Rupp

Resume: LI Consequence of missing LI in linear systems of

equations:Rows < Columns

equations of system thefor solution a be Let 00

3

2

1

4

3

2

1

3431

1411

≠

=

x

bbb

xxxx

aa

aa

( )

*

11 14

* *

31 34

0 0 *

due to linear dependency in the column vectors, a 0 must exist so that0

0 .0

Then we must also have that: 0 .Thus, an infinite amount of solut

xa a

x A xa a

Ax A x x b bα

≠

= =

= + = + =

ions exists for this system of equations.


Markus Rupp

System Identification Problem: The impulse response hk of a

linear time-invariant system is to estimate based on observations of input and output signals. What conditions does the input skhave to satisfy to ensure a uniqueidentification?

hsk rk


Markus Rupp

System Identification Solution:

11 2 1 1

2

1

0

MM M

M

hs s s s rh

hh

−−

−

=


Markus Rupp


11 2 1 1

22 3 1 2

1

0

MM M

MM M

hs s s s rhs s s s r

hh

−−

−+

=


Markus Rupp


For a unique identification, the rows (columns) need to be linearly independent!

11 2 1 1

22 3 1 2

11 1

01 2 2 2 1

MM M

MM M

M M

M M M M M

hs s s s rhs s s s r

hs rhs s s s r

−−

−+

− −

+ − −

=


Markus Rupp

System Identification A signal with such a property is said

to be of persistent excitation (Ger.: hartnäckige Anregung).

Question: is a complex-valued exponential harmonic sk=exp(jΩοk) of persistent excitation?

What about a sinusoid sk=sin(Ωοk)?

Persistent Excitation Consider excitation signal

only system of order 1 (constant) can be identified.

[ ][ ] 1

322

)1(201

,...,,,,...,,,

xeeeeexeeeex

es

ooooo

oooo

o

jMjjjj

Mjjjj

kjk

ΩΩΩΩΩ

−ΩΩΩΩ

Ω

==

=

=

Univ.-Prof. Dr.-Ing. Markus Rupp111

Persistent Excitation Consider

excitation signal:

only system of order 2 can be identified.

( )[ ][ ]

( )

b

j

a

j

jkjjkjok

ba

Mjjjjb

Mjjjja

kjkjok

xj

exj

ex

eeeej

ks

xj

xj

x

eeeexeeeex

eej

ks

oo

oooo

oooo

oooo

oo

22

21))1(sin(

21

21

,...,,,,...,,,

21)sin(

2

1

1

)1(20

)1(20

Ω−Ω

Ω−Ω−ΩΩ+

−Ω−Ω−Ω−Ω−

−ΩΩΩΩ

Ω−Ω

−=

−=+Ω=

−=

=

=

−=Ω=


Persistent Excitation How many signals (frequencies Ωo) of the

form exp(jΩok) are required to identify a time-invariant system of order M?

How many signals (frequencies Ωo) of the form sin(Ωok) are required to identify a time-invariant system of order M?



Markus Rupp

Vector Spaces Definition 2.20: Let T be a set of vectors in a

vector space S over a set of scalars R(C,Q,Z,N,B). The set of vectors V that can be reached by all possible (finite) linear combinations of vectors in T is the span (Ger.: aufgespannte Menge, erzeugte Menge, lineare Hülle) of the vectors. :

)span(TV =

Vector Spaces Note that this is an elegant way of saying something in short

that otherwise would require a lengthy formulation. If T is a set, then span(T) is a set as well!

The number field is typically K=R or K=C. The span is typically defining a vector space!


Markus Rupp

∈∈==

==

∈+==

=

∑=

21

22

2

11

1

,|)span(

,....,2,1|,|)span(

,

TxKxTV

NixTKyxTV

yxT

iiii

N

i

i

αα

βαβα


Markus Rupp

Vector Spaces Definition 2.21: Let S be a vector space, and let

T be a set of vectors from S such that span(T)=S. If the elements in T are linearly independent, then T is said to be a Hamel basis for S.

Example 2.34: The vectors p1=[1,6,5], p2=[-2,4,2], p3=[1,1,0] and p4=[7,5,2] are linearly dependent. Note that T=p1,p2,p3 spans the space R3 and thus is a basis for R3.

Example 2.35: The vectors p1=[1,0,0], p2=[0,1,0] and p3=[0,0,1] are linearly independent and are a basis for R3. This basis is often called natural basis (Ger.: natürliche Basis).

Georg Karl Wilhelm Hamel (12 September 1877–4 October 1954)

http://en.wikipedia.org/wiki/September_12

http://en.wikipedia.org/wiki/1877

http://en.wikipedia.org/wiki/October_4

http://en.wikipedia.org/wiki/1954


Markus Rupp

Vector Spaces Example 2.36: Consider the (3,2) single parity

check code in GF(2)3 with the elements:V=[000],[011],[101],[110].A Hamel basis is given by:

=]011[]101[

G


Markus Rupp

Example 2.37 The following 3x3 matrices are a Hamel basis in R3x3:

100000000

,010000000

,001000000

000100000

,000010000

,000001000

000000100

,000000010

,000000001


Markus Rupp

Example 2.38 Are these 3x3 matrices a Hamel basis?:

Answer: These matrices are not a basis for R3x3. However, the are a basis for the subspace in R3x3,

that has a zero row and column sum!

−−

−−

−

−

−

−

110110000

,101101000

,110000110

,101000101


Markus Rupp

Vector Spaces Definition 2.22: The number of elements in a set

is its cardinality |A| (Ger.: Kardinalität).

Theorem 2.1: If two sets T and U are Hamel bases for the same vector space S, then T and U are of the same cardinality.

Proof: (only for the finite dimensional case). Letbe two bases for S and with at least one coefficient unequal to zero,

,...,,;,...,,2121 nm

qqqUpppT ==

mm pcpcpcq +++= ...22111

4,1,1,1,1 =−−+−−+= AA :QPSK jjjj

Georg Cantor (3.3.1845-6.1.1918), German Mathematician

Presenter

Presentation Notes

|Z|<|[0,1]|, set has unique elements, multiset or bag can have multiple elements


Markus Rupp

Vector Spaces say c1, then: Thus

is also a basis for S. Further substitution leads to

( ),...,,

...1

21

2211

1

m

mm

ppq

pcpcqc

p −−−=

,,...,,,

,,...,,,...

,...,,,

,...,,,

1321

1321

321

321

mm

mm

m

m

qqqqq

pqqqq

ppqq

pppq

−

−


Markus Rupp

Vector Spaces Consequently, we must have: Now starting with q1 instead of p1, we

obtain Thus, we must have n=m.

Definition 2.23: Let T be a Hamel basis for S. The cardinality of T is the dimension of S, |T|=dim(S). It equals the number of linearly independent vectors, required to span the space S.

nm ≥

.mn ≥


Markus Rupp

Vector Spaces Note: each vector space has at least one

Hamel basis.

Operations in the vector space are often simpler and of lower complexity in their basis.


Markus Rupp

Example 2.33 revisited Consider blind channel estimator (blind

equalization):

h1

h2

sk

g1

g2

sk(1)

sk(2)

-0

rk(1)

rk(2)


Markus Rupp

Example 2.33 revisited Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

(1)]T

When are the vectors linearly dependent?

In order to obtain linear dependency, we must have a sufficient number of vectors r1

(1)..rm(1), r1

(2)..rm(2), so that

2m>N since the vectors rk(1,2) of dimension N can maximally

span RN!

[ ] mN

m

mmm

mmmm

Rg

g

gg

g

rrrrrr

rgrgrgrgrgrg

2

2

21

1

11

)2()2(2

)2(1

)1()1(2

)1(1

)2(2

)2(222

)2(121

)1(1

)1(212

)1(111

R;0R,...,,,,...,,

......

×∈==

−

−

+++=+++

Presenter

Presentation Notes



Markus Rupp

Inner Products Definition 2.24: A vector space for which an

inner vector product is defined is said to be a pre- Hilbert space (Ger.: Innerer Vektorproduktraum, Vor-Hilbertraum).

An inner vector product maps two vectors onto one scalar with the following properties:

zyzxzyx

yxyx

xyyx

xxx

,,,)3

,,)2

,,)1

else. 0 and 0for 0,)0*

+=+

=

=

≠>

αα


Markus Rupp

Inner Products Example 2.39: Consider the space of continuous functions

C[a,b] with the two elements x(t) and h(t). Let x(t) be an input signal and h(t) the impulse response of a low pass. We have:

Example 2.40: Consider vectors in C: Matrices can also build inner products:

∫

∫

==

−=

T

T

gxdgx

dThxTy

0

0

,)()(

)()()(

τττ

τττ

BAABH ,)Trace( =

xyyx H=,

Presenter

Presentation Notes

When is trace a norm? Sqrt(tr(A^HA))norm!


Markus Rupp

Inner Products Example 2.41: Consider the expectation of a

random variable:

This is also an inner vector product, however with an additional weighting function.

[ ]

f

dxdyyxfxy

yx,

),(xyE xy

=

= ∫∫

Inner Products Consider two 2-dimensional vectors x and y. Their inner product (projection) is a

measure of non-orthogonality:


Markus Rupp

000 <=> y,xy,xy,xy

xy

xy

x


Markus Rupp

Complete Space Remember:

Definition 2.11: A metric Space (X,d) is called complete (Ger.: vollständig), if every Cauchy-sequence converges.

Presenter

Presentation Notes

112-114 muss weiter nach hinten!!! Wegen normierung


Markus Rupp

Hilbert- and Banach Spaces Definition 2.25: A complete normed

(metric) vector space is said to be a Banach space. Is there additionally an inner vector product (the norm is an induced norm), the space is said to be a Hilbert space.

Example 2.42: The space of continuous functions (C[a,b],d∞) is a Banach space.

Example 2.43: The space of continuous functions (C[a,b],dp) is for finite p not a Banach space since it is not complete.

Stefan Banach, (30.3.1892-31.8.1945) Polish Mathematician


Markus Rupp

Hilbert- and Banach Spaces Example 2.44: The space of sequences lp(0,∞) is a

Banach space. For p=2 it is also a Hilbert space. Example 2.45: The space of functions Lp[a,b] is a

Banach space. For p=2 it is also a Hilbert space.This Hilbert space is often denoted as L2(R) for functions and l2(R) for sequences.

In the following part of the lecture we will exclusively consider Hilbert spaces. (if not noticed otherwise)


Markus Rupp


Markus Rupp

Orthogonality Definition 2.26: Vectors of a pre-Hilbert space

are said to be orthogonal or perpendicular (Ger.: normal) if <x,y>=0.

Definition 2.27: Vectors of a pre-Hilbert space are said to be orthonormal if:

1,

1,

0,

=

=

=

yy

xx

yx


Markus Rupp

Orthogonality Example 2.46: The following set of

vectors is orthogonal:

How do we have to modify the set, in order to make the vectors orthonormal?

−−

−−

−

−

1111

,

1111

,

1111

,

1111

Orthogonality Consider a set of orthogonal vectors.

We then have:

Consider a set of orthonormal vectors. We then have:


Markus Rupp

ii

m

ii

m

iii

m

iii pppp ,,

1

2

11∑∑∑

===

= ααα

∑∑∑===

=m

ii

m

iii

m

iii pp

1

2

11, ααα


Markus Rupp

Orthogonality Example 2.47: The following set of functions is

orthonormal in [-π, π]:1,ejt/4, ej2t/4, ej3t/4

The inner product is defined as:

∫

∫

∫

−

−

−

=−=

−=

==

=

π

π

π

π

π

π

π

π

π

0)4/exp(21

)4/2exp()4/exp(21,

:)4/2exp()( and )4/exp()(Let

)()(21, *

dtjt

dtjtjtgf

jttgjttf

dttgtfgf


Markus Rupp

Vector Spaces At this point several interesting questions arise:

Under which conditions is the linear combination of vectors unique?

Which is the smallest set of vectors required to describe every vector in S by a linear combination?

If a vector x can be described by a linear combination of pi ;i=1..m, how do we get the linear weights ci ;i=1..m?

Of which form do the vectors pi ;i=1..m need to be in order to reach every point x in S?

If x cannot be described exactly by a linear combination of pi ;i=1..m, how can it be approximated in the best way (smallest error)?


Markus Rupp

Vector Spaces Definition 2.28: A Hamel basis of dimension m is

said to be orthogonal if for all basis vectors T=p1,p2,p3...,pm:

Definition 2.29: A Hamel basis of dimension m is said to be orthonormal if all basis vectors T=p1,p2,p3...,pm:

=≠≠

=jiji

ppji ;0

;0,

ji

ji jiji

pp

−=

=≠

=

δ;1;0

,


Markus Rupp

Orthogonality Let T=p1,p2,...,pn be a set of vectors. How can we

find a set S=q1,q2,...,qm with m smaller or equal to n so that span(S)=span(T) and the vectors in S are orthonormal?

Gram-Schmidt Method: 1) take p1 and construct

2) build:

3) continue:

4) if ei=0, throw away pi+1.

Erhard Schmidt 13.1.1876- 6.12.1959 German Mathematician

2223

22311332

111211221

1111

,/

;,,

,/;,

,/

eeeq

qqpqqppe

eeeqqqppe

pppq

=

−−=

=−=

=

http://de.wikipedia.org/wiki/1876

http://de.wikipedia.org/wiki/6._Dezember



Markus Rupp

Orthogonality Advantages of orthogonal bases:

[ ]

[ ] . of tionmultiplicaleft by Proof

basis lorthonorma an be let

T

T

ppp

pf

pf

pf

ppp

ppfppfppff

pppPcbaf

321

3

2

1

321

332211

321

,,

,

,

,

,,

,,,

,,],,,[

=

++=

==


Markus Rupp

Vector Spaces Definition 2.30: If there are two bases,

that span the same space with the additional property:

then these bases are said to be dual or biorthogonal (biorthonormal for ki,j=1).

,...,,;,...,,2121 mm

qqqUpppT ==

jijijikqp −= δ,,


Markus Rupp

Vector Spaces Example 2.48: let:

These pairs build a dual basis in R2. Consider

then:

[ ] [ ] [ ] [ ] ;1,0;1,1;1,1;0,12121

TTTT qqpp ==−==

Tbaf ],[=

212211

212211

)(,,

)(,,

qbaqaqpfqpff

pbpbapqfpqff

+−+=+=

++=+=


Markus Rupp

Resume Space Metric Set vs. Vector complete

Convergence Cauchy-Sequence

Linear vector space Subspace


Markus Rupp

Resume Consider the following function

in C[-∞,∞]:[ ]

[ ]

( )?,C[- in converge it Doesto? converge sequence this does Where

: for sequence functional the Compute

p

n

n

d

xxnn

xf

/n

xxxf

],

2exp

/21)(

1

21exp

21)(

20

2

2022

∞∞

−−=

=

−−=

π

σ

σπσ

( )?,L[- in converge it Does pd],∞∞

x

-50 -40 -30 -20 -10 0 10 20 30 40 50

f n(x

)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

n=2

n=10


Markus Rupp

Resume Cardinality of a set of QPSK?

Dimension of a vector space?= cardinality of the Hamel basis

What is the dimension of the vectors that form a Hamel basis for QPSK ?

What is a linear combination? When are vectors linearly independent?

4,1,1,1,1 =−−+−−+= AA :QPSK jjjj


Markus Rupp

Resume00 0

1 01 1

1 02 2

1 1 0

2

1 1 01 1

1 1 0

1 1 0

1 0

2 1

1 0

......

M

K

MK K

MK K

k N M

M

k M

k

k M

hy xh hy x

h hy xh h h

xh h hy x

h h hy x

y h hh h

y hy hy h

−

−

−− −

−

− + −

−

− −

−

−

=

=

2

2

1

1 0...

k N M

k

k

k

kNk

x

xx

h xy H x

− − +

−

−

=

Which column vectors of these matrices are LI?

Presenter

Presentation Notes

Obere matrix K+1 Vektoren mit K+1 Elem. sind LU, untere:N+M-1 Vektoren mit N-1 El. Kann nicht LU sein


Markus Rupp

Example 2.33 revisited Consider blind channel estimator (blind

equalization):

h1

h2

sk

g1

g2

sk(1)

sk(2)

-0

rk(1)

rk(2)


Markus Rupp

Example 2.33 revisited Observe N receive values at each sensor:

rk(1)..=[rk

(1),rk-1(1)…,rk-N+1

(1)]T

When are the vectors linearly dependent?

In order to obtain linear dependency, we must have a sufficient number of vectors r1

(1)..rm(1), r1

(2)..rm(2), so that

2m>N since the vectors rk(1,2) of dimension N can maximally

span RN!

[ ] mN

m

mmm

mmmm

Rg

g

gg

g

rrrrrr

rgrgrgrgrgrg

2

2

21

1

11

)2()2(2

)2(1

)1()1(2

)1(1

)2(2

)2(222

)2(121

)1(1

)1(212

)1(111

R;0R,...,,,,...,,

......

×∈==

−

−

+++=+++

Presenter

Presentation Notes


Example 2.33 revisited It turns out in fact that the following three

conditions are sufficient for a solution. 2m>N We now understand that we require to have

persistent excitation of the input signal sk. The Bezout condition must be satisfied for the

channels: no common zero! The solution itself we will provide later in the

context of SVD methods.


Markus Rupp


Markus Rupp

Resume The Hilbert space is a linear vector

space with the following three properties: 1) 2) 3)

Complete (Vollständig)MetricNormed (Normiert)There exists an inner product


Markus Rupp

Norms Definition 2.31: Let S be a vector space with

elements x. A real-valued function ||x|| is said to be a norm (length) of x, if the following four properties are satisfied:

Note that just as for metrics 0) follows from 1) and 3)! (can you show this property)?

)inequality (triangle )3

Cfor)2

0 ifonly and if 0)1

every for 0)0

yxyx

xx

xx

Sxx

+≤+

∈=

==

∈≥

ααα


Markus Rupp

Norms Note, norms and metrics are related

concepts:

Example 2.49:

pppp xxdyxyxd =−= )0,(;),(

ii

n

ii

n

ii

xxl

xxl

xxl

max1

2

22

111

=−

=−

=−

∞∞

=

=

∑

∑

:norm

:norm

:norm

[ ] )(sup)(

)()(

)()(

,

2

22

11

txtxL

dttxtxL

dttxtxL

bat

b

a

b

a

∈∞∞ =−

=−

=−

∫

∫

:norm

:norm

:norm

Norms Why is l2 never smaller than l00?

With this method you can prove that


Markus Rupp

22

max 221max

1 ....ii

i i

xx x x x xx ∞ ∞

≥

= = = + ≥∑ ∑

1/ pp

n x x x∞ ∞

≥ ≥


Markus Rupp

Norms Definition 2.32: A normed, linear space is a

pair (S,||.||) in which S is a vector space and ||.|| is a norm on S.

Metrics of a normed linear space are defined by norms.

Definition 2.33: A vector is said to be normalized (Ger.: normiert, unit vector, Ger.: Einheitsvektor) if ||x||=1.

Except of the zero vector all vectors can be normalized.


Markus Rupp

Norms Note: The elements (vectors) of a linear

and normed space are not necessarily normalized vectors.

Only the space is normed, meaning that a norm exists for this space.

Norms Consider a sequence of vectors yk and a

limit y*. To evaluate whether yk converges to the

limit, we employ a norm:

Or, equivalently we can substitute xk=yk-y* and obtain:


Markus Rupp

0lim

0lim *

=

=−

∞→

∞→

kk

kk

x

yy


Markus Rupp

Norms Norm Equivalence Theorem 2.2: Let ||.|| and

||.||‘ be two norms on finite dimensional spaces Rn

(Cn,Qn,Zn,Nn,Bn), then we have:

Proof: It is to show that Without restricting generality we set (Ger.: oBdA)

||.||‘= ||.||2 and obtain for the lower bound:

0'lim,0lim == ∞→∞→ kkkk xx if only and if

2

21

21 1

1

1max

xx

xexexexx

exx

n

ii

n

i

n

iiiii

n

iii

≤

=≤≤≤

=

∑∑ ∑

∑

== =

=

α

α

:Thus

0,;' >≤≤ βαβα xxx

Presenter

Presentation Notes

Beweis einfacher als im Moon stirling, ohne chauchy schwarz, cauchy folgen sind die konsequenz!


Markus Rupp

Norms For the upper bound β consider the case:

And we obtain:

With the above condition, x cannot be the zero vector and since ||.||>0 there must be a positive lower bound β:

Since the property is true for the l2 norm, it is also true for all other norms!

02

>= cx

0;122

2

>≥= ββ

xxxxx

xxc β≤=2

'1'1' xxxxxxαβ

βα ≤≤→≤≤


Markus Rupp

Norms Note that the equivalence theorem is defined in terms of

Cauchy sequences. This is a consequence from the proof. Since if a particular norm tends to zero, then it does for every norm. The equivalence is to be understood in this sense.

Norms often find their application in terms of energy relations. It can be in form of average energy (l2-norm) or peak values (l∞-norm).

Thus, norms appear often when we describe systems. (Robustness, nonlinear systems, convergence of learning systems such as adaptive equalizers or controllers).


Markus Rupp

Norms Example 2.50: Hands-free telephone,

Ger.: Freisprechtelefon

+Far end speaker sk rk

Local speaker vk

h -

h

2

2hh − :energyerror System


Markus Rupp

Norms Example 2.51: Consider an adaptive equalizer as

part of a receiver.

The equalizer has a cost function of the form

which it tries to minimize (adaptive process). The error at its output is given by an error vector:

h f+sk rkvk kk ss ≈ˆ

( )kk ssf −ˆ

],...,[]ˆ,...,ˆ,ˆ[ 11 kNkkkNkNkNkNkk eesssssse −+−+−−− =−−−=

Presenter

Presentation Notes

Bild, kostenfunktion, adaptivität


Markus Rupp

Norms Example 2.51: Let the adaptive equalizer have

the property that the error signal from k-1 to k is mapped via the learning rule: y=g(x)=x3. Under which condition is the equalizer adaptive?

( )

1......sup

...

...supsup?

21

21

61

61

21

21

22

2

21

2

2

31

31

1

<++++

=++++

=

==

=

−−−

−−−

−−−

−

−∈

−

−−

−

−

Nkk

Nkk

Nkk

Nkk

k

kle

k

Nk

k

k

Nk

k

eeee

eeee

ee

e

eeg

e

ee

hk

For |ek|<1 the error term at the output is always smallerthan at the input.

Presenter

Presentation Notes

Fehlerwerte müssen im Betrag kleiner eins sein!


Markus Rupp

Norms Example 2.52: Consider the linear system in matrix-

vector-form.

Consider the ratio of input and output energy. When do we have a passive system, when an amplifier?

1 1 0 2

1 0

2 1 2

1 0 1

1 0

1,,

......

...

k N M k N M

M

k M k

k k

k M k

N M kNN k

y h h xh h

y h xy h xy h h xy H x

− + − − − +

−

− − −

− −

−

+ −

=

=

)(sup?

2

2,1

2

2,N

kMN

kNlx Hf

x

yhk

=−+

∈

Norms on Matrices In principle we can consider matrices

as vectors and apply the same norms, e.g., apply the l2 norm:

and obtain the Frobenius norm.


Markus Rupp

),min(

...)trace( 222

21

1 1

22

nmp

AAAA pH

m

i

n

jijF

=

++=== ∑∑= =

σσσ

Ferdinand Georg Frobenius (1849-1917)

http://de.wikipedia.org/wiki/Ferdinand_Georg_Frobenius

Norms on Matrices Similarly

define norms on matrices


Markus Rupp

ijij

m

i

n

jij

AA

AA

max'

1 1

'

1

=

=

∞

= =∑∑


Markus Rupp

Induced Norms Inner vector products are helpful for defining

norms. For example, the (quadratic) l2 norm can be written as inner vector product:

This is thus said to be an induced (Ger.: induzierte) norm. The norm is induced by the inner vector product.

Of the lp norms for vectors only the l2 norm is an induced norm!

xxx

xxxxxx n

,

,...

2

222

21

2

2

=

=+++=


Markus Rupp

Induced Norms Example 2.53: induced norm in L2[a,b]

Note that for real-valued signals we have:

How can this be formulated for complex-valued signals?

yxyxyx ,22

2

2

2

2

2=−−+

21

221

2)()(),()(

== ∫

b

a

dttxtxtxtx


Markus Rupp

Induced Norms Vector norms can also be used to induce norms on

multivariate functions. Consider the following example:

( )2,0

expsup

)exp(sup

),(sup

)exp(),(

22

21

2121],[

2

2121],[2

2

21],[

212121

21

21

21

≈+

−=

−=

−

=

−=

=

=

=

xx

xxxx

xxxxx

f

lx

xxff

xxxxxxf

xxx

xxx

xxx

normSelect


Markus Rupp

Induced Norms

-5-4

-3-2

-10

12

34

5

-5-4

-3-2

-10

12

34

50

0.05

0.1

0.15

0.2

0.25

0.3

0.35


Markus Rupp

Induced Vector Norms Matrix-norms are typically induced

vector-norms. Consider matrix A

Note that (spectral norm):

This is the largest singular value.

px

ppx

p

pxpp

xAxxA

x

xAAA

p 100ind,supsupsup =≠≠ =

===

( )AAAxAA Hx maxmax21ind,2

)(sup2

λσ === =

nmR ×∈

Induced Vector Norms


Markus Rupp


Markus Rupp

Spectral Norm Consider firstly a vector with M elements, based on the

sequence exp(jΩk):

In other words, all these vectors are linearly dependent , independent of their dimension M and frequency Ω.

If, however, M different frequencies are selected, M linearly independent vectors can be obtained.

[ ][ ]

[ ] kMT

mkM

kMT

kM

TkM

xmjmMkjmkjmkjx

xjMkjkjkjx

Mkjkjkjx

,,

,1,

,

)exp())1(exp()),...,1(exp()),(exp(

...)exp())2(exp(),...,exp()),1(exp(

))1(exp()),...,1(exp(),exp(

Ω=++−Ω+−Ω+Ω=

Ω=+−ΩΩ+Ω=

+−Ω−ΩΩ=

+

+


Markus Rupp

Spectral Norm Consider now the spectral norm for the vector xM,k:

0 1

0

1,

0

0exp( ( ))

...exp( ( 1))exp( ( ))

(exp( )) exp( ) exp(

(exp( ))(exp( )) exp( ( 1))(exp( )) exp( ( ))

M

MM k l

h hj k l

hhAx

j k l mj k l m n

h

H j j l

H jH j j l mH j j l m n

−

−+

Ω +

= Ω + + − Ω + + +

Ω Ω = = Ω Ω Ω + − Ω Ω + +

,

,

22,ind

2

, 2, 2,ind

, 2

)

(exp( ))exp( ( 1))exp( ( ))

sup

sup sup (exp( ))M k

M l

x M

M k l

x M MM k l

j l

H j xj l mj l m n

AxA

x

AxH j A

x

=

+Ω= Ω →∞

+

Ω = Ω Ω + − Ω + +

=

= → Ω =

Further Matrix Norms Recall:

Why is that? Consider an element yi of y=Ax


Markus Rupp

,ind,ind11

,ind

11

1

111

and:BUT

:generalin

max:maxrow

max:maxcolumn

∞∞

=≤≤∞

=≤≤

==

≠

==

=

∑

∑

AAAA

AA

AAA

AA

pp

Tn

jijmi

m

iijnj

?maxmax1

11

==

=→=

∑

∑∑

=

==

m

jjijxix

m

jjiji

m

jjiji

xAy

xAyxAy

jj

We can maximize the expression if sign(xj)=sign(Aij) as then xjAij =|xjAij|

If we further restrict the elements of vector x, e.g., ||x||=1, then there are two possibilities


Markus Rupp

,ind,ind11and

∞∞== AAAA

[ ]

[ ] ijj

jijT

ijj

jijj

T

AxAxx

AxAxx

maxmaxand10,...0,1,0,...,0,0

maxand11,...,1,1

1==→=

==→±±±=

∑

∑∑∞


Markus Rupp

,ind,ind11and

∞∞== AAAA

11

1

1111

1,1

111

11,

maxmaxmax

maxmaxmax

maxmaxmaxmax

maxmaxmaxmax

11

AAAxA

xAxAxxA

A

AAAxA

xAxAxxA

A

m

iijnj

iijj

jjij

ij

jjij

ixxxind

n

jijmi

jiji

jjijxi

jjijixxxind

====

===

====

===

∑∑∑∑

∑∑

∑∑∑

∑

=≤≤

==

∞=

≤≤=

=∞=∞

∞∞

∞

∞∞


Markus Rupp

Submultiplicative Property Some of the matrix norms (all p-norms) satisfy

the submultiplicative property:

Note that this property holds for arbitrary matrices, for example

But in this case the p-norms are different functions!

∞=≤ ,...2,1;ind,ind,ind,

pBAABppp

4332 , ×× ∈∈ RBRA

pppxByAxAB ≤

Presenter

Presentation Notes

Also auch für vektoren Ax=y

Submultiplicative Property Proof:


Markus Rupp

indpindpp

pxindp

p

pindpx

p

p

p

px

p

pxindp

BAx

xBA

x

yA

x

y

y

yA

xxAB

AB

p

p

pp

,,1,

,1

11,

sup

sup

supsup

==

≤

==

=

=

==


Markus Rupp

Submultiplicative Property Not all matrix norms have this submultiplicative

property: Example 2.54:

∆∆∆

∆

>

==

=

BAAB

BA

AA ij

:have we For 1111

max

Submultiplicative Property A similar property, however, is true

for all unitarily invariant norms:

unitarily invariant norm: (C is unitary)


Markus Rupp

ind

ind

ABAB

BAAB

,2

,2

≤

≤

xxC =


Markus Rupp

Matrix Norms Example 2.55: Consider a causal impulse response hk.

In which case is hk an allpass (amplifier)?

Obviously, the energy of the input sequence must be identical to the energy of the output sequence (for every sequence).

Note that the vectors xN+M-1,k and yN,k have different length!

1 1 0 2

1 0

2 1 2

1 0 1

1 0

1,,

......

...

k N M k N M

M

k M k

k k

k M k

N M kNN k

y h h xh h

y h xy h xy h h xy H x

− + − − − +

−

− − −

− −

−

+ −

=

=


Markus Rupp

Matrix Norms We must thus have for an allpass:

This in turn requires that N∞. For the spectral norm we have at least:

1

supsup

limlimlim

,

,1

,

0,1

,1

0,

,1,1,

==

==

==

−+≠

−+

−+

≠

−+∞→−+∞→∞→

indpN

pkMN

pkN

x

pkMN

pkMNN

xindpN

pkMNNpkMNNNpkNN

H

x

y

x

xHH

xxHy

least at havemust we

have we Since

( ) ( )indNN

jj HeHeH,2

limmax ∞→Ω

Ω∞

Ω ==


Markus Rupp

Matrix Norms An allpass is thus difficult to describe by

matrices of finite dimension or by the spectral norm.

The reason for the finite dimension is that the initial values need also to be considered to reflect the lossless property of the allpass. This property can be required over the entire observation horizon.


Markus Rupp

Matrix Norms Allpasses are better suited for state space forms. Note:

Example 2.56: yk=xk-2

=

+=

+=

+

+

k

kT

k

k

kkT

k

kkk

xz

dcbA

yz

dxzcy

xbzAz

1

1

[ ]

=

+=

+

=

+

+

k

k

k

k

kkk

kkk

xz

yz

xzy

xzz

00110001000,1

10

0010

1

1


Markus Rupp

Matrix Norms Consider the

instantaneous energy:1

2 2 2 2 2 21 2 1 0 1 1

2 2 2 2 2 21 1

2 2 2 2 2 21 2 1 1 1

2 2 2 2 2 21 1

2 2 2 21 0

1 1

0 1 00 0 11 0 0

k k

k k

k k k k k k

k k k k k k

N N N N N NN N

N k kk k

z zy x

z z y z z x

z z y z z x

z z y z z x

z z y z z x

z y z x

+

+ −

+ + + + +

+ −

+= =

=

+ + = + +

+ + = + +

+ + = + +

+ + = + +

+ = +∑ ∑

⇐=+

++1

2

2

2

0

2

2

2

1

N

NN

xz

yz

For all N


Markus Rupp

Matrix Norms Allpass as system with feedback

dcbA

q-1

1+kzkz

kx ky


Markus Rupp

Matrix Norms This more general form also allows for describing allpasses

by linear time-variant systems. Example 2.57: Let b=sqrt(1-a2); 0<a<1

.a-1bfor allpass an also describes

allpass. an describes

2kk =

=

−

=

=

−

=

k

kkk

k

kk

kk

k

k

kk

k

k

k

A

xHxx

abba

yy

A

xHxx

abba

yy

)2(

)1(

)2(

)1(

)2(

)1(

)2(

)1(


Markus Rupp

Matrix Norms Allpass

unitary! be to said are conditions these satisfythat , Matrices vectors arbitrary allfor

:havemust we allpass anfor this tocontrast In

have we systems,invariant -timelinear For

N

kMN

pkMN

pkMNN

pkMN

pkN

pkMN

pkN

x

pkMN

pkMNN

xindpN

Hx

x

xH

x

y

x

y

x

xHH

,1

,1

,1

,1

,

,1

,

0,1

,1

0,

1

supsup

−+

−+

−+

−+

−+≠

−+

−+

≠

==

==

Matrix Norms Let us summarize:

The 2 induced matrix norm

tells us whether a system is passive or active

If a matrix is unitary, it describes an allpass.


Markus Rupp


)(sup2

λσ === =

2,ind

2,ind

1

1

A amplifier

A attenuator

> ⇒

< ⇒


Markus Rupp

Resume Norms are…

Metrics in a linear vector space. A norm defined over a linear vector space makes it to a

normed, linear vector space. The knowledge of one norm is in general sufficient due

to the Equivalence Theorem

An induced norm is… a norm, that is build by a „simpler construct“

(example: the inner vector product induces the p=2 norm)

0'lim,0lim == ∞→∞→ kkkk xx if only and if


Markus Rupp

Resume An induced matrix p-norm is…

a vector p-norm, applied to a matrix.

The spectral norm is… the vector p=2 norm induced to a matrix. In case of linearly time-invariant systems, it is identical with the

maximum magnitude of the transfer function.

Why is this true? Due to submultiplicative property of vector norms.

The submultiplicative property for matrices reads… .

pindppxAxA

,≤

BAAB ≤


Markus Rupp

Induced Vector Norms (Resume) Matrix-norms are typically induced

vector-norms. Consider matrix A

Note that (spectral norm):

This is the largest singular value.

px

ppx

p

pxpp

xAxxA

x

xAAA

p 100ind,supsupsup =≠≠ =

===


)(sup2

λσ === =

nmR ×∈


Markus Rupp

Resume Inner vector products can also be written in terms

of induced norms. In R:

In C:

yxyxyx ,22

2

2

2

2

2=−−+

yxyxyx

yxyxyx

,Re4

,Re22

2

2

2

2

2

2

2

2

2

=−−+

=−−+


Markus Rupp

Small Gain TheoremGeorge Zames(1934-1997) in 1966

Consider the following system with feedback.

Is it stable?

1

0,5q-1

yNhN

gNzN

xN

-

Pole at-0,5

stable!

http://en.wikipedia.org/wiki/George_Zames


Markus Rupp

Small Gain Theorem Consider the following system with feedback.

Is it stable?

y=1/(1+h2)

0,5q-1

yNhN

gNzN

xN

-


Markus Rupp

Small Gain Theorem The „Small Gain Theorem“ can be interpreted as

an extension from linear systems to nonlinear systems for the stability problem.

In linear time-invariant systems the gain of the open loop is responsible for stability (BIBO stability).

The Small Gain Theorem can be applied to non-linear as well as to time-variant systems.


Markus Rupp

Small Gain Theorem Consider for this the input output relation of a

system described by vectors: Definition 2.34: A mapping H is said to be lp-

stable if there exists two positive constants β,γ so that for all input sequences the following is true:

Definition 2.35: The smallest positive value γ, for which lp-stability is obtained, is said to be the gain of H (Ger.: Verstärkung).

Note: BIBO (Bounded Input Bounded Output) stability is equivalent to l∞- stability.

βγ +≤=pNpNNpN

xxHy

NNNxHy =


Markus Rupp

Small Gain Theorem Consider now a system with feedback, comprising

of two systems G and H and corresponding gains γgand γh described by the following signals:

[ ][ ]

NNNNNN

NNNNNN

yuGgGz

zxHhHy

+==

−==


Markus Rupp

Small Gain Theorem

HN

GN +

yNhN

gNzN uN

xN

-

Presenter

Presentation Notes

Zames: control theorist, born in poland, prof in canada (mc gill)


Markus Rupp

Small Gain Theorem Theorem 2.3: (Small Gain Theorem) If both gains

γg and γh are such that:

then the two signals gN and hN of the system with feedback are bounded by:

1<hgγγ

[ ]

[ ]ghhNhNhg

N

hggNgNhg

N

xug

uxh

βγβγγγ

βγβγγγ

+++−

≤

+++−

≤

11

11


Markus Rupp

Small Gain Theorem Proof: Start with:

the norm is given by:

The derivation for gN is analogue.

[ ]

[ ]hggNgNhg

hggNgNNhg

ghNhNgN

gNgN

NNNN

ux

uxh

hux

gx

gGxh

βγβγγγ

βγβγγγ

ββγγ

βγ

+++−

=

++++=

++++≤

++≤

+≤

11

NNNNNN yugzxh +=−= ;

HN

GN +

yNhN

gNzN uN

xN

-


Markus Rupp

Example 2.58 Let the automatic control

of a cell phone power amplifier look like this:

H2

G2 +

y2h2

g2z2 u2=0

x2

-

=

+

+=

37,031

01

5,2

120

2

212

212

221

221

2

G

xx

xx

H

ρρ

ρρ


Markus Rupp

Example 2.58 Is this control stable? We describe G2 and H2 by their gains:

2222

,2

22,2222

222

,2

212

212

221

221

22,2222

37,037,0

31

5,20

15,2

120

gggGz

hh

xx

xx

hHy

ind

ind

ind

ind

=

=≤

=

+

+=≤

ρρ

ρρ


Markus Rupp

Example 2.58

stable.-l thus ,0,372,5:gain loop open the for have thus We

21

37,09137,0

sup37.0

31

5,21

21

5,2

sup

01

5,2

120

sup0

15,2

120

22

21

22

21

2

1

,2

22

21

21

2

212

2122

2

2

221

221

22

2

2

1

212

212

221

221

,2

212

212

221

221

22

21

2

12

<×=

=+

+=

=+

++

+=

+

+

=

+

+

=+

=

γ

ρρ

ρρ

ρρ

ρρ

ρρ

ρρ

xx

xx

xx

xx

xx

xx

h

xx

xx

xx

xx

xx

xx

ind

xx

h

ind


Markus Rupp

Example 2.60 Consider the following problem:

Let the shape of the signal s(t) be known at the receiver.

How is the filter g(t) to design so that the SNR at the output of g(t) is maximum?

matched filter (Ger.: Signalangepasstes Filter)Dwight O. North 1943

g(t)+s(t)

v(t)d(t)

r(t)


Markus Rupp

Chauchy-Schwarz Inequality Theorem 2.4: In an inner-product space S in Cn

with induced norm ||.|| we have:

The inequality becomes an equality if and only if x=αy.

Proof: We start with the simple fact that

22, yxyx ≤

02

≥− yx α


Markus Rupp

Chauchy-Schwarz Inequality We find further that

The minimum is obtained for: Thus:

−

−+−=

+−=−≤

2

2

*

*2

2

2

22

2

2

2

2

2

2

22

2

2

2

,,,

,Re20

y

yx

y

yxy

y

yxx

yyxxyx

αα

ααα

2

2

,

y

yx=α

2

2

2

2

2

2

2

,min0

y

yxxyx −=−≤ αα

22, yxyx ≤→


Markus Rupp

Chauchy-Schwarz Inequality We are not done!

For x=αy we obtain ||x-αy||=0. Due to the norm property , equality to zero

can only be obtained for || x-αy||=0 if the argument is zero.

Thus equality is obtained if and only if x=αy.

0≥x


Markus Rupp

Chauchy-Schwarz Inequality The Cauchy-Schwarz inequality implies:

for S in Cn:

for the space of real-valued functions in the interval [a,b]:

( )( )yyxxyx HHH ≤2

∫∫∫ ≤

b

a

b

a

b

a

dttgdttfdttgtf )()()()( 22

2


Markus Rupp

Chauchy-Schwarz Inequality Example 2.59 Correlation coefficient

Remember: For the correlation coefficient we have:

Proof: Consider zero mean random variablesx‘=x-mx and y‘=y-my:

( )( )[ ]1

E111 ≤≤−⇒≤≤−

yx

yxxy

-y-xσσ

mmr

( )( )[ ]yx

yxxy

-y-xσσ

mmr

E=

[ ] ( )( )[ ]

[ ] [ ][ ] [ ]

'

EEEE

EE

xy22yx

yx

yx

yx

yxxy

y'x'y'x'y'x'

yxxy

r

mmmmr

===

−−=

−=

σσ

σσσσ


Markus Rupp

Chauchy-Schwarz Inequality Chauchy-Schwarz requires that

[ ] [ ] [ ][ ] [ ] [ ]

[ ][ ] [ ] 1

y'Ex'Ey'x'E

y'Ex'Ey'x'E

y'Ex'Ey'x'E

''','

22

22y'x'

222

22

22

≤=

≤

≤

≤

r

yxyx


Markus Rupp

Still Example 2.59 Correlation Coefficient Note that E[.] defines an ensemble average. In practical problems this is often replaced by a

time average (ergodic processes). Cauchy Schwarz still holds then:

( )( )

( ) ( )∑∑

∑

==

=

==

=

N

k

N

k

N

k

mN

mN

mmNr

1

22

1

22

1

1;1

1

ykyxkx

yx

ykxk

xy

-y-x

-y-x

σσ

σσ


Markus Rupp

Back to Example 2.60 Matched Filter (Ger.: Signalangepasstes Filter):

g(t)+s(t)

v(t)d(t)

r(t)

∫∫∫∫

∫

∫

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

=−≤

−==

−=

ττττττττ

τττ

τττ

dgdrdtgdr

dtgrttd

dtgrtd

)()()()(

)()()(

)()()(

2222

2

02


Markus Rupp

Example 2.60 Matched filter:

g(t)+s(t)

v(t)d(t)

r(t)

)()()()()(

""

)()()( 222

ταταττατ

ττττ

−≈−=⇔=−

=→

≤ ∫∫∞

∞−

∞

∞−

srgrg

dgdrtd

:if only and if Identity,tpoint time suitable a to energyreceiver Maximize 0


Markus Rupp

Example 2.60 Graphical:

r(τ)g(t-τ)

∫∞

∞−

−= τττ dtgrtd )()()(

d(t)

g(t-τ)

τ

t


Markus Rupp

Alternative Example 2.60

g(t)+as(t)

v(t)d(t)

r(t)[ ] [ ]

)(),()(),(')'(

')'()'()()()(

)(),()()()()(

).()()(;1

2222

2

2

2

22

22

ττσττσττσ

τττττδτστττ

ττττττττ

τδστ

ggtgtgdtg

dtgdtgdtg

stgdstgdstg

tt(t) s(t)

vvv

v

v

vEN

:zero is signal if energy Receiver

aES

:zero is noise if energy ReceivervvEaE

with vanda signals random Consider

=−−=−=

−−−=

−=

−=

−=

−=

=+=

∫

∫ ∫∫

∫∫


Markus Rupp

Alternative Example 2.60

g(t)+As(t)

v(t)d(t)

r(t)

2v

2v

)()(2v

2

1)(),(

2v

20

1)(),()(),()(),(

maxmax

:SNR Maximize

)(),(:only noise todueenergy Receiver

)(),()(

:only signal todueenergy Receiver

σσττ

ττσττ

ττσ

ττ

ταττ ==−

=

=

−==

=−=

ssggstg

NS

ggN

stgttS

stgstg


Markus Rupp

Example 2.61 A fixed symbol sequence sk in C with L symbols is

transmitted for the purpose of synchronization at the beginning of a TDMA frame.

Design an optimal synchronization filter fk. Received sequence without noise:

s Infodata ik s …[ ],...i,,i, 21 ssrT =


Markus Rupp

Example 2.61 Correlation filter:

Because of Chauchy-Schwarz we have:

kkk

kkkkkkf

L

lllkk

srf

ffrrfr

frd

k

αα ≈=

=

= ∑−

=+

:thus

,,,max2

1

0

*

Example 2.62Werner Karl Heisenberg (5.12.1901 – 1.2.1976)Hermann Klaus Hugo Weyl, (9.11.1885 – 8.12.1955)

Time-frequency uncertainty:

for F(jω) being the Fourier transform of f(t).


Markus Rupp

21

)(21

)(21

)(

)(

2

22

2

22

≥

∫

∫

∫

∫∞

∞−

∞

∞−∞

∞−

∞

∞−

ωωπ

ωωωπ

djF

djF

dttf

dtttf

21

)(

)(

)(

)(

2

22

2

22

≥

∫

∫

∫

∫∞

∞−

∞

∞−∞

∞−

∞

∞−

ωω

ωωω

djF

djF

dttf

dtttf

Example 2.62 Recall the following properties:

1)

2)

3)

(partial integration with Dirichlet condition that f(t)0 for t+-oo)


Markus Rupp

∫∫∞

∞−

∞

∞−

=−

=−

=

dttfdttfttf

djFdtfjt

jFjdt

tfd

n

nn

nn

n

F

F

)()()(2

)()()(

)()()(

2'

ωω

ωω

Peter Gustav Lejeune Dirichlet13. 2.1805 in Düren; † 5.5.1859 in Göttingen)

Presenter

Presentation Notes

Düren ist in nordrhein westfalen am nordrand der eifel


http://de.wikipedia.org/wiki/D%C3%BCren


http://de.wikipedia.org/wiki/G%C3%B6ttingen

Example 2.62 Cauchy Schwarz says


Markus Rupp

22

2

2 2

2

2 2 2

1 ( )2

( ) ( ) ( ) ( )

( ) '( ) ( ) ' ( )

f t dt

f t g t dt f t dt g t dt

tf t f t dt t f t dt f t dt

∞

−∞

∞ ∞ ∞

−∞ −∞ −∞

∞ ∞ ∞

−∞ −∞ −∞

= − ∫

≤

≤

∫ ∫ ∫

∫ ∫ ∫

Example 2.62


Markus Rupp

∫

∫

∫=

∫∫≤

≤

∫−

∞

∞−

∞

∞−∞

∞−

∞

∞−

∞

∞−

∞

∞−∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∫

∫∫

∫∫

ωωπ

ωωωπ

djF

djF

dttf

dttft

dttf

dttf

dttf

dttft

dttfdttftdttf

2

22

2

22

2

2

2

22

2222

2

)(21

)(21

)(

)(

)(

)('

)(

)(

41

)(')()(21

Plancherel

Michel Plancherel (* 16.1.1885; † 4.3.1967 in Zürich) was a Suisse mathematician.

Presenter

Presentation Notes

Parseval is about Fourier series, while Plancherel as about Fourier transform (continuous)!

http://de.wikipedia.org/wiki/16._Januar



http://de.wikipedia.org/wiki/Z%C3%BCrich

Example 2.62 Time-frequency uncertainty:

Q: for which functions do we obtain equality?

A: tf(t)=f‘(t)=exp(-αt2)


Markus Rupp

21

)(

)(

)(

)(

2

22

2

22

≥

∫

∫

∫

∫∞

∞−

∞

∞−∞

∞−

∞

∞−

ωω

ωωω

djF

djF

dttf

dtttf

Alternative form (M.Vetterli et al. 2013) Define 1st and 2nd order moment:


Markus Rupp

Also works in the sequence domain:


Markus Rupp

Signal Processing 1 - nt.tuwien.ac.at · Resume Consider symmetric impulse response: h 0 =a, h 1...

Documents

Transcript of Signal Processing 1 - nt.tuwien.ac.at · Resume Consider symmetric impulse response: h 0 =a, h 1...