2D DSP2D DSP

Nuno VasconcelosUCSD

Images• the incident light is collected by an image sensor

– that transforms it into a 2D signal E

θPo

V

θ2Pi

V

2

2D-DSP• in summary:

– image is a N x M array of pixels– each pixel contains three colors– overall, the image is a 2D discrete-space

signal– each entry is a 3D vector

}0{},...,0{),,,(],[ 121

MnNnbgrnnx ∈=

– for simplicity, we consider only singlechannel images

},...,0{2 Mn ∈

b hi d l i i h f d

},...,0{},...,0{],,[

2

121

MnNnnnx

∈∈

3

– but everything extends to color in a straightforward manner

Separable sequences• a trivial concept,

– but probably the only real novelty in this lecture– very important in practice, because it reduces 2D problem to

collection on 1D problems

• Definition: a sequence is separable if and only if

][][][ ngnfnnx ×=

where f[.] and g[.] are 1D functions

][][],[ 2121 ngnfnnx ×=

• note: there are many examples of separable sequences• but most sequences are not separable

4

but most sequences are not separable

Linear Shift Invariant (LSI) systems• straightforward extension of LTI systems

• Definition: a system T that maps x[n1,n2] into y[n1,n2] isLSI if and only if– it is linearit is linear

{ }{ } { }],[],[

],[],[

212211

212211

nnxbTnnxaTnnbxnnaxT

+==+

– it is shift invariant

{ } { }],[],[

],[],[

212211

212211

nnbynnay +=

{ } ],[],[ 22112211 mnmnymnmnxT −−=−−

5

2D convolution• the operation

][][][ knknhkkxnny ∑ ∑∞ ∞

is the 2D convolution of x and h

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑−∞= −∞=

– we will denote it by

],[],[],[ 212121 nnhnnxnny ∗=

• this is of great practical importance:– for an LSI system the response to any input can be obtained by– for an LSI system the response to any input can be obtained by

the convolution with this impulse response– the IR fully characterizes the system

it is all that I need to measure

6

– it is all that I need to measure

2D convolution• has various properties of interest• but these are the ones that you have already seen in 1D

(check handout)• some of the more important:

commutative:– commutative:

– associative:

xyyx ∗=∗

( ) ( )zyxzyx ∗∗=∗∗

– distributive:

l ti ith i l

( ) zxyxzyx ∗+∗=+∗

– convolution with impulse:

],[],[],[ 2211221121 mnmnxmnmnnnx −−=−−∗ δ

7

2D convolution• as in 1D, it is most easily done in graphical form• e.g. how do we convolve these two sequences?

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑∞

−∞=

∞

−∞=

n2 n2

(3) (4)

n1 n1(1) (2)*

(3) (4)

],[ 21 nnx ],[ 21 nnh

8• we need four steps

2D convolution• step 1): express sequences in terms of (k1, k2)

][][][ kkhkk∑ ∑∞ ∞

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑−∞= −∞=

k2 k2

(3) (4)

k1 k1(1) (2)

],[ 21 kkx ],[ 21 kkh

9

2D convolution• step 2): invert h(k1, k2)

][][][ kkhkk∑ ∑∞ ∞

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑−∞= −∞=

2k2

(1)(2)

k2

(3) (4)

k2

(1) (2)

k1

( )( )

(3)(4)

k1(1) (2) k1

(1) (2)

(3) (4)1

],[],[ 2121 kkhkkg −−=],[ 21 kkh ],[ 21 kkh −

10

2D convolution• step 3): shift g(k1, k2) by (n1, n2)

][][][ kkhkk∑ ∑∞ ∞

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑−∞= −∞=

k2

(1)(2)

k2

(1)(2)

(3)(4)

n2

this sendswhatever is at (0,0) to (n1,n2)

k1

( )( )

(3)(4)

k1n1

],[],[ 2121 kkhkkg −−=][

],[ 2211

knknhnknkg =−−

11

],[ 2211 knknh −−

2D convolution• step 4): point-wise multiply the two signals and sum

][][][ kkhkk∑ ∑∞ ∞

– e.g. for (n1,n2) = (1,0)

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑−∞= −∞=

1 2

k2k2 k2

k1

(1)(2)

(3)(4)

n2

k1

x =k1

n2

][ knknh −−

n1

][ kkx ][ nny

n1

12

],[ 2211 knknh],[ 21 kkx ],[ 21 nny

2D convolution• step 4): point-wise multiply the two signals and sum

][][][ kkhkk∑ ∑∞ ∞

– e.g. for (n1,n2) = (2,0)

],[],[],[ 221121211 2

knknhkkxnnyk k

−−= ∑ ∑−∞= −∞=

1 2

k2k2 k2

k1

(1)(2)

(3)(4)

n2

k1

x =k1

n2

(3)

][ knknh −−

n1

][ kkx ][ nny

n1

13

],[ 2211 knknh],[ 21 kkx ],[ 21 nny

etc.

2D convolution• is this the only way to look at convolution?

– any signal can be written as

e g

],[],[],[ 221121211 2

knknkkxnnxk k

−−= ∑ ∑∞

−∞=

∞

−∞=

δ

– e.g.n2 n2 n2

n1

(a) (b)

n1

(a)

n1

(b)= +

],1[],[ 2121 nnbnna −+ δδ

14]0,0[x ]0,1[x

2D convolution• we combine this

][][][ knknkkxnnx ∑ ∑∞ ∞

δ

with the properties of the convolution

],[],[],[ 221121211 2

knknkkxnnxk k

−−= ∑ ∑−∞= −∞=

δ

– commutative:

– associative:

xyyx ∗=∗

( ) ( )zyxzyx ∗∗=∗∗associative:

– distributive:

( ) ( )zyxzyx ∗∗=∗∗

( ) zxyxzyx ∗+∗=+∗

– convolution with impulse:

],[],[],[ 2211221121 mnmnxmnmnnnx −−=−−∗ δ

15• to obtain another interpretation

2D convolution• it is done like this

],[*],[],[ 212121 nnhnnxnny =

],[*],[],[

][][][

21221121

212121

1 2

nnhknknkkx

y

k k⎟⎟⎠

⎞⎜⎜⎝

⎛−−= ∑ ∑

∞

−∞=

∞

−∞=

δ

( )],[*],[],[],[ 21221121211 2

1 2

nnhknknkkxnnyk k

k k

−−=

⎠⎝

∑ ∑∞

−∞=

∞

−∞=

∞ ∞

δ

• note that this is just our definition of convolution1 2k k∞ ∞

][][][ knknkkxnnx ∑ ∑∞ ∞

δ

(no surprises here)

],[],[],[ 221121211 2

knknkkxnnxk k

−−= ∑ ∑−∞= −∞=

δ

16

( p )• but we want to think about it like this, not like this

2D convolution• we can think of

( )][*][][][ hkkkk∑ ∑∞ ∞

δ

as the following sequence of operations

( )],[*],[],[],[ 21221121211 2

nnhknknkkxnnyk k

−−= ∑ ∑−∞= −∞=

δ

1. set y[n1, n2] = 0, for all (n1,n2)2. for each (k1,k2) such that x[k1,k2] is not zero

• set α = x[k k ]• set α = x[k1,k2]• shift h[n1,n2] by (k1,k2)• multiply the entire sequence by α

• add the entire sequence to y[n1,n2]

( )],[*],[],[ 21221121 nnhknknnnz −−= δα

17],[],[],[ 212121 nnznnynny +=

2D convolutionn2 n2

• examplen2 n2

*(3) (4)

n1 n1(1) (2)

• (k1,k2) = (0,0)],[ 21 kkx ],[ 21 nnh

n2 n2

(3) (4)

n2

(3) (4)

n1= n1(1) (2)+ 1xn1(1) (2)

18],[ 21 nny ],[ 2211 knknh −−],[ 21 nny

2D convolutionn2 n2

• examplen2 n2

*(3) (4)

n1 n1(1) (2)

• (k1,k2) = (1,0)],[ 21 kkx ],[ 21 nnh

n2n2

(3) (7)

n2

(3) (4) (3) (4)(4)

= n1+ 1xn1(1) (3) n1(1) (2) (1) (2)(2)

19],[ 2211 knknh −−],[ 21 nny ],[ 21 nny

2D convolutionn2 n2

• examplen2 n2

*(3) (4)

n1 n1(1) (2)

• (k1,k2) = (0,1)],[ 21 kkx ],[ 21 nnh

n2

(4) (9) (4)

n2

(3) (7) (4)

n2

(1) (2)

(3) (4)(3) (4)

=n1(1) (3) (2) n1(1) (3) (2) n1+ 1x

(1) (2)

20],[ 21 nny ],[ 21 nny ],[ 2211 knknh −−

2D convolutionn2 n2

• examplen2 n2

*(3) (4)

n1 n1(1) (2)

• (k1,k2) = (1,1)],[ 21 kkx ],[ 21 nnh

n2

(4) (10) (6)

n2

(3) (7)

n2

(4) (9) (4)

(3) (4)

(1) (2)

(3) (4)(4)

=n1(1) (3) (2) n1+ 1xn1(1) (3) (2)

(1) (2)

21],[ 21 nny ],[ 2211 knknh −−],[ 21 nny

2D convolution• note the differences with the previous convolution

– before we were computing one y[n1,n2] at a time

k1

k2

(1)(2)n2

k1

k2

x = k1

k2

n2(3)

k1

(3)(4)

],[ 2211 knknh −−

n1

k1

],[ 21 kkx

k1

],[ 21 nny

n1

– now we update the entire sequence at a time],[ 2211 knknh],[ 21 kkx ],[ 21 nny

n2 n2

(3) (7)

n2

(3) (4) (3) (4)(4)

=n1(1) (3)

(4) (10)

(2)

(6)

n1+ 1x

(3) ( )

n1(1) (3)

(4) (9)

(2)

(4)

(3) ( )

(1) (2)

(3) (4)( )

22],[ 21 nny ],[ 2211 knknh −−],[ 21 nny

Separable systems• Definition: a system is separable if and only if its

impulse response is a separable sequence

• note that, in this case the convolution simplifies

][][],[ 221121 nhnhnnh ×=

note that, in this case the convolution simplifies

∑ ∑∞

−∞=

∞

−∞=

−−=1 2

],[],[],[ 22112121k k

knknhkkxnny

∑ ∑∞

−∞=

∞

−∞=

−−=1 2

][][],[ 22211121k k

knhknhkkx

∑ ∑∞

∞

−∞=

∞

−∞=

−−=1 2

][],[][ 22221111k k

knhkkxknh

23∑∞

−∞=

−=1

],[][ 21111k

nkfknh

Separable systems• the convolution simplifies to

∑∞

][][][ nkfknhnny

with

∑−∞=

−=1

],[][],[ 2111121k

nkfknhnny

∞

• note that:

∑∞

−∞=

−=2

][],[],[ 2222121k

knhkkxnkf

• note that:– for a fixed k1, f[k1,n2] is 1D convolution of x[k1,n2] and h2[n2]

][][][ nhnkxnkf ∗

– for a fixed n2, y[n1,n2] is 1D convolution of f[n1,n2] and h1[n1]

][],[],[ 222121 nhnkxnkf ∗=

24][],[],[ 112121 nhnnfnny ∗=

Separable systems• the convolution simplifies to a sequence of 1D steps• step1) for every k1,

– f[k1,n2] is 1D convolution of x[k1,n2] and h2[n2]

][],[],[ 222121 nhnkxnkf ∗=

– which means: “convolve the columns of x with h2 to obtain columns of f”

n22n2

*

n1h2

25

n1

],[ 21 nnx ],[ 21 nnf

Separable systems• step2) for every n2,

– y[n1,n2] is 1D convolution of f[n1,n2] and h1[n1]

][],[],[ 112121 nhnnfnny ∗=

– which means: “convolve the rows of f with h1 to obtain rows of y”

n2n2

*

n1n1 h1

26

],[ 21 nny],[ 21 nnf

Separable systems• in summary, if we have a separable system

][][],[ 221121 nhnhnnh ×=

to convolve with x[n1,n2] we:

][][],[ 221121

– 1) “convolve the columns of x with h2 to create f”

][],[],[ 222121 nhnkxnkf ∗=

– 2) “convolve the rows of f with h1 to obtain y”

][][][ nhnnfnny ∗ ][],[],[ 112121 nhnnfnny ∗=

27

The Discrete-Space Fourier Transform• as in 1D, an important concept in linear system analysis

is that of the Fourier transform• the Discrete-Space Fourier Transform is the 2D

extension of the Discrete-Time Fourier Transform2211][)( ωω ωω eennxX njnj∑∑ −−

2121

2211

2211

1 2

)(1][

],[),(

ωωωω

ωω

ωω ddeeXnnx

eennxX

njnj

jj

n n

∫∫

∑∑

=

=

• note that this is a continuous function of frequency

2121221 ),()2(

],[ ωωωωπ

ddeeXnnx ∫∫=

q y• the nomenclature distinguishes it from the 2D Discrete

Fourier transform (we will get back to this)h d h DSFT f i l k lik ?

28

• what does the DSFT of an image look like?

Image spectrum• two images, the magnitude, and phase of their FTs

29

Phase and Magnitude

• curious fact– all natural images have about the same magnitude transformall natural images have about the same magnitude transform– monotonically decaying with frequency

1)( ωω ∝X

• hence, phase seems to matter, but magnitude largely d ’t

22

21

21 ),(ωω

ωω+

∝X

doesn’t• we can see this through the following experiment• take two pictures swap the phase transforms compute• take two pictures, swap the phase transforms, compute

the inverse• here is what you get

30

The importance of phaseReconstruction with zebra

phase, cheetah magnitudeReconstruction with cheetah

phase, zebra magnitude

31

LSI systems• why do we care so much about Fourier transforms?• note that when we convolve a sequence with a complex

exponential,

2211],[ 21njnj eennx ϖϖ=

we get

][][][ knknxkkhnny −−=∞ ∞

∑ ∑

e e ],[

],[],[],[

)(j)(j21

22112121

222111

1 2

ϖϖkkh

knknxkkhnny

knkn

k k

=

=

−−∞ ∞

−∞= −∞=

∑ ∑

∑ ∑

)(][),(e e 21

jj

21

2211

1 2

ϖϖϖϖϖϖ

HnnxHnn

k k

==

−∞= −∞=∑ ∑

32

),(],[ 2121 ϖϖHnnx=

LSI systems• but we have seen that, for an LSI system

– the output in response to x[n1,n2]– is the convolution with the impulse response h[n1,n2]– hence, the response to

2211][ njnj eennx ϖϖ

– is

2211],[ 21jj eennx =

),(],[],[ 212121 ϖϖHnnxnny =

– this means that complex exponentials are the eigenfunctions of LSI systems

)(][][ 212121y

– when we input an eigenfunction, we get back the same function– but scaled by H(ω1,ω2)– this is called the frequency response of the system

33

q y p y

LSI systems• this is remarkable, since

– we know that any signal can be represented as a weighted sum of comple e ponentialsof complex exponentials

21212211

1 2

],[),( ωω ωω eennxX njnj

n n∑∑= −−

21212212211

1 2

),()2(

1],[ ωωωωπ

ωω ddeeXnnx njnj∫∫=

– when the signal is fed to an LSI system, each exponential isscaled by H(ω1,ω2)

– hence the frequency response completely characterizes the– hence, the frequency response completely characterizes the system

– and the DSFT of the output is just the product of the two

34

),(),(),( 212121 ωωωωωω XHY =

Example• the system with impulse response on the left• has frequency response

n2

⎟⎠⎞

⎜⎝⎛

61

⎟⎠⎞

⎜⎝⎛

31

n1

⎠⎝ 6

⎟⎠⎞

⎜⎝⎛

61

⎟⎠⎞

⎜⎝⎛

61

⎟⎠⎞

⎜⎝⎛

61

⎠⎝ 3

2121

11111

],[),( 2211

1 2

ωω ωω= −−∑∑ njnj

n neennhH

11161

61

61

61

31 2121 ϖϖϖϖ ++++= −− jjjj eeee

• note that:

21 cos3

cos33

ϖϖ ++=

– the response is 1 at DC– lower for high frequencies

this system is a low pass filter

35

– this system is a low-pass filter

WARNING• WARNING, WARNING, WARNING!• the equivalence

11 ϖϖ jnjn−

1cos2

11

ϖϖϖ

nee jnjn

=+

• is the oldest trick in the DSP book!

2n2

• please do not fall for it– you can “read” the sequence that has this DSFT

111

2

⎟⎠⎞

⎜⎝⎛

61

⎞⎛

⎟⎠⎞

⎜⎝⎛

31

by applying this trick and the definition of DSFT!

2121 cos31cos

31

31 ),( ϖϖωω ++=H n1⎟

⎠⎞

⎜⎝⎛

61

⎟⎠⎞

⎜⎝⎛

61

⎟⎠⎞

⎜⎝⎛

61

36

WARNING• Quizz: which on the left is the DSFT of

this image?

?

37

WARNING• the way to think about this is the

following:

– this image has low frequencies– this image has low frequencieshorizontally (ω1)

– high frequencies vertically (ω2)the spectrum is a delta function along– the spectrum is a delta function along(ω1) and harmonics along (ω2)

– the spectrum is this“b i i h i l b ”

38

– wrong way: “because image is horizontal spectrum must be too”

57