Numerical Analysis Fast Fourier Transform

39
Numerical Analysis Fast Fourier Transform Andreas Kloeckner h[email protected]i Base slides c Michael T. Heath CS 450 · December 6, 2013

Transcript of Numerical Analysis Fast Fourier Transform

Page 1: Numerical Analysis Fast Fourier Transform

Numerical AnalysisFast Fourier Transform

Andreas Kloeckner 〈[email protected]〉Base slides c© Michael T. Heath

CS 450 · December 6, 2013

Page 2: Numerical Analysis Fast Fourier Transform

Goal today:Think about data in the frequency domain.

Page 3: Numerical Analysis Fast Fourier Transform

Audio synthesis demo

What about cosines?

Page 4: Numerical Analysis Fast Fourier Transform

Audio synthesis demo

What about cosines?

Page 5: Numerical Analysis Fast Fourier Transform

Complex Exponential NotationDiscrete Fourier Transform

Fast Fourier TransformApplications

Complex Exponential NotationWe will use complex exponential notation based on Euler’sidentity

eiθ = cos θ + i sin θ

where i =√−1

Since e−iθ = cos(−θ) + i sin(−θ) = cos θ − i sin θ, we have

cos(2πkt) =e2πikt + e−2πikt

2

and

sin(2πkt) = ie−2πikt − e2πikt

2

Pure cosine or sine wave of frequency k is equivalent tosum or difference of complex exponentials of halfamplitude and frequencies k and −k

Michael T. Heath Scientific Computing 4 / 32

Page 6: Numerical Analysis Fast Fourier Transform

Roots of unityDiscrete Fourier Transform

Fast Fourier TransformApplications

Roots of UnityFor given integer n, primitive nth root of unity is given by

ωn = cos(2π/n)− i sin(2π/n) = e−2πi/n

nth roots of unity, called twiddle factors in this context, aregiven by ωkn or by ω−k

n , k = 0, . . . , n− 1

< interactive example >Michael T. Heath Scientific Computing 5 / 32

Page 7: Numerical Analysis Fast Fourier Transform

Synthesis with roots of unitydemo

Page 8: Numerical Analysis Fast Fourier Transform

Trigonometric InterpolationDiscrete Fourier Transform

Fast Fourier TransformApplications

Trigonometric Interpolation

In modeling periodic or cyclic phenomena, sines andcosines are more appropriate functions than polynomialsor piecewise polynomials

Representation as linear combination of sines and cosinesdecomposes continuous function or discrete data intocomponents of various frequencies

Representation in frequency space may enable moreefficient manipulations than in original time or spacedomain

Michael T. Heath Scientific Computing 3 / 32

Page 9: Numerical Analysis Fast Fourier Transform

Audio demo

Page 10: Numerical Analysis Fast Fourier Transform

Questions

Why is there odd distortion in the audio?What is the cost of applying that matrix?Can we do this more cheaply?Can we do also apply the inverse more cheaply?

Page 11: Numerical Analysis Fast Fourier Transform

DFTDiscrete Fourier Transform

Fast Fourier TransformApplications

Discrete Fourier Transform

Discrete Fourier transform, or DFT, of sequencex = [x0, . . . , xn−1]

T is sequence y = [y0, . . . , yn−1]T given

by

ym =n−1∑

k=0

xk ωmkn , m = 0, 1, . . . , n− 1

In matrix notation, y = Fn x, where entries of Fouriermatrix Fn are given by

{Fn}mk = ωmkn

For example,

F4 =

1 1 1 11 ω1 ω2 ω3

1 ω2 ω4 ω6

1 ω3 ω6 ω9

=

1 1 1 11 −i −1 i1 −1 1 −11 i −1 −i

Michael T. Heath Scientific Computing 6 / 32

Page 12: Numerical Analysis Fast Fourier Transform

Inverse DFTDiscrete Fourier Transform

Fast Fourier TransformApplications

Inverse DFT

Note that

1

n

1 1 1 11 ω−1 ω−2 ω−3

1 ω−2 ω−4 ω−6

1 ω−3 ω−6 ω−9

1 1 1 11 ω1 ω2 ω3

1 ω2 ω4 ω6

1 ω3 ω6 ω9

=

1 0 0 00 1 0 00 0 1 00 0 0 1

In general,F−1n = (1/n)FH

n

Inverse DFT is therefore given by

xk =1

n

n−1∑

m=0

ym ω−mkn k = 0, 1, . . . , n− 1

DFT gives trigonometric interpolant using onlymatrix-vector multiplication, which costs only O(n2)

Michael T. Heath Scientific Computing 7 / 32

Page 13: Numerical Analysis Fast Fourier Transform

More DFTDiscrete Fourier Transform

Fast Fourier TransformApplications

DFT, continued

DFT of sequence, even purely real sequence, is in generalcomplex

Components of DFT y of real sequence x of length n areconjugate symmetric : yk and yn−k are complex conjugatesfor k = 1, . . . , (n/2)− 1

Two components of special interest arey0, whose value is sum of components of x, is sometimescalled DC component, corresponding to zero frequency(i.e., constant function)yn/2, corresponding to Nyquist frequency, which is highestfrequency representable at given sampling rate

Components of y beyond Nyquist frequency correspond tofrequencies that are negatives of those below Nyquistfrequency

Michael T. Heath Scientific Computing 8 / 32

Page 14: Numerical Analysis Fast Fourier Transform

ExampleDiscrete Fourier Transform

Fast Fourier TransformApplications

Example: DFT

For cyclic sequence x,

F8 x = F8

1−11−11−11−1

=

00008000

= y

Sequence has highest possible rate of oscillation (between1 and −1) for this sampling rate

In transformed sequence, only nonzero component is atNyquist frequency (in this case y4)

Michael T. Heath Scientific Computing 10 / 32

Page 15: Numerical Analysis Fast Fourier Transform

Computing the DFT IDiscrete Fourier Transform

Fast Fourier TransformApplications

Computing DFT

By taking advantage of symmetries and redundancies indefinition of DFT, shortcut algorithm can be developed forevaluating DFT very efficiently

For illustration, consider case n = 4

From definition of DFT

ym =3∑

k=0

xk ωmkn , m = 0, . . . , 3

Writing out four equations in full

y0 = x0ω0n + x1ω

0n + x2ω

0n + x3ω

0n

y1 = x0ω0n + x1ω

1n + x2ω

2n + x3ω

3n

y2 = x0ω0n + x1ω

2n + x2ω

4n + x3ω

6n

y3 = x0ω0n + x1ω

3n + x2ω

6n + x3ω

9n

Michael T. Heath Scientific Computing 11 / 32

Page 16: Numerical Analysis Fast Fourier Transform

Computing the DFT IIDiscrete Fourier Transform

Fast Fourier TransformApplications

Computing DFT, continuedNoting that

ω0n = ω4

n = 1, ω2n = ω6

n = −1, ω9n = ω1

n

and regrouping, we obtain four equations

y0 = (x0 + ω0nx2) + ω0

n(x1 + ω0nx3)

y1 = (x0 − ω0nx2) + ω1

n(x1 − ω0nx3)

y2 = (x0 + ω0nx2) + ω2

n(x1 + ω0nx3)

y3 = (x0 − ω0nx2) + ω3

n(x1 − ω0nx3)

DFT can now be computed with only 8 additions orsubtractions and 6 multiplications, instead of expected(4− 1) ∗ 4 = 12 additions and 42 = 16 multiplications

Actually, even fewer multiplications are required for thissmall case, since ω0

n = 1, but we have tried to illustrate howalgorithm works in general

Michael T. Heath Scientific Computing 12 / 32

Page 17: Numerical Analysis Fast Fourier Transform

Computing the DFT IIIDiscrete Fourier Transform

Fast Fourier TransformApplications

Computing DFT, continued

Main point is that computing DFT of original 4-pointsequence has been reduced to computing DFT of its two2-point even and odd subsequences

This property holds in general: DFT of n-point sequencecan be computed by breaking it into two DFTs of halflength, provided n is even

General pattern becomes clearer when viewed in terms offirst few Fourier matrices

F1 = 1, F2 =

[1 11 −1

], F4 =

1 1 1 11 −i −1 i1 −1 1 −11 i −1 −i

, . . .

Michael T. Heath Scientific Computing 13 / 32

Page 18: Numerical Analysis Fast Fourier Transform

Computing the DFT IV

Xk =N−1∑

n=0

xne− 2πiN nk

Xk =

N/2−1∑

m=0

x2me− 2πiN (2m)k +

N/2−1∑

m=0

x2m+1e− 2πiN (2m+1)k

Xk =

N/2−1∑

m=0

x2me− 2πiN/2 mk

︸ ︷︷ ︸DFT of even−indexed part of xm

+ e− 2πiN k

N/2−1∑

m=0

x2m+1e− 2πiN/2 mk

︸ ︷︷ ︸DFT of odd−indexed part of xm

= Ek + e− 2πiN kOk .

Page 19: Numerical Analysis Fast Fourier Transform

Summary: (Power-of-2) algorithmDiscrete Fourier Transform

Fast Fourier TransformApplications

FFT Algorithm

procedure fft(x, y, n, ω)if n = 1 then

y[0] = x[0]else

for k = 0 to (n/2)− 1p[k] = x[2k]s[k] = x[2k + 1]

endfft(p, q, n/2, ω2)fft(s, t, n/2, ω2)for k = 0 to n− 1

y[k] = q[k mod (n/2)]+ωkt[k mod (n/2)]

endend

{ bottom of recursion }

{ split into even andodd subsequences }

{ call fft procedurerecursively }

{ combine results }

Michael T. Heath Scientific Computing 16 / 32

Complexity?

Page 20: Numerical Analysis Fast Fourier Transform

Summary: (Power-of-2) algorithmDiscrete Fourier Transform

Fast Fourier TransformApplications

FFT Algorithm

procedure fft(x, y, n, ω)if n = 1 then

y[0] = x[0]else

for k = 0 to (n/2)− 1p[k] = x[2k]s[k] = x[2k + 1]

endfft(p, q, n/2, ω2)fft(s, t, n/2, ω2)for k = 0 to n− 1

y[k] = q[k mod (n/2)]+ωkt[k mod (n/2)]

endend

{ bottom of recursion }

{ split into even andodd subsequences }

{ call fft procedurerecursively }

{ combine results }

Michael T. Heath Scientific Computing 16 / 32Complexity?

Page 21: Numerical Analysis Fast Fourier Transform

(Power-of-2) algorithm: Remarks IDiscrete Fourier Transform

Fast Fourier TransformApplications

FFT Algorithm, continued

There are log2 n levels of recursion, each of which involvesO(n) arithmetic operations, so total cost is O(n log2 n)

For clarity, separate arrays were used for subsequences,but transform can be computed in place using no additionalstorage

Input sequence is assumed complex; if input sequence isreal, then additional symmetries in DFT can be exploited toreduce storage and operation count by half

Output sequence is not produced in natural order, buteither input or output sequence can be rearranged at costof O(n log2 n), analogous to sorting

Michael T. Heath Scientific Computing 17 / 32

Page 22: Numerical Analysis Fast Fourier Transform

(Power-of-2) algorithm: Remarks IIDiscrete Fourier Transform

Fast Fourier TransformApplications

FFT Algorithm, continued

FFT algorithm can be formulated using iteration rather thanrecursion, which is often desirable for greater efficiency orwhen using programming language that does not supportrecursion

Despite its name, fast Fourier transform is an algorithm,not a transform

It is particular way of computing DFT of sequence inefficient manner

< interactive example >

Michael T. Heath Scientific Computing 18 / 32

Page 23: Numerical Analysis Fast Fourier Transform

ComplexityDiscrete Fourier Transform

Fast Fourier TransformApplications

Complexity of FFT

DFT is defined in terms of matrix-vector product, whosestraightforward evaluation would appear to require O(n2)arithmetic operations

Use of FFT algorithm reduces work to only O(n log2 n),which makes enormous practical difference in timerequired to transform large sequences

n n log2 n n2

64 384 4096128 896 16384256 2048 65536512 4608 262144

1024 10240 1048576

Michael T. Heath Scientific Computing 19 / 32

Page 24: Numerical Analysis Fast Fourier Transform

Inverse FFTDiscrete Fourier Transform

Fast Fourier TransformApplications

Inverse Transform

Due to similar form of DFT and its inverse (only sign ofexponent differs), FFT algorithm can also be used tocompute inverse DFT efficiently

Ability to transform back and forth quickly between timeand frequency domains makes it practical to performcomputations or analysis that may be required inwhichever domain is more convenient and efficient

Michael T. Heath Scientific Computing 20 / 32

Page 25: Numerical Analysis Fast Fourier Transform

LimitationsDiscrete Fourier Transform

Fast Fourier TransformApplications

Limitations of FFT

FFT algorithm is not always applicable or maximallyefficientInput sequence is assumed to be

Equally spacedPeriodicPower of two in length

First two of these follow from definition of DFT, while thirdis required for maximal efficiency of FFT algorithm

Care must be taken in applying FFT algorithm to producemost meaningful results as efficiently as possible

For example, transforming sequence that is not reallyperiodic or padding sequence to make its length power oftwo may introduce spurious noise and complicateinterpretation of results < interactive example >

Michael T. Heath Scientific Computing 21 / 32

Page 26: Numerical Analysis Fast Fourier Transform

What about n = pq?Discrete Fourier Transform

Fast Fourier TransformApplications

Mixed-Radix FFT

Mixed-radix FFT algorithm does not require number ofpoints n to be power of two

More general algorithm is still based on divide andconquer, with sequence being split at each level bysmallest prime factor of length of remaining sequence

Efficiency depends on whether n is product of small primes(ideally power of two)

If not, then much of computational advantage of FFT maybe lost

For example, if n itself is prime, then sequence cannot besplit at all, and “fast” algorithm becomes standard O(n2)matrix-vector multiplication

Michael T. Heath Scientific Computing 22 / 32

Cooley-Tukey FFT algorithm

Page 27: Numerical Analysis Fast Fourier Transform

What about n = pq?Discrete Fourier Transform

Fast Fourier TransformApplications

Mixed-Radix FFT

Mixed-radix FFT algorithm does not require number ofpoints n to be power of two

More general algorithm is still based on divide andconquer, with sequence being split at each level bysmallest prime factor of length of remaining sequence

Efficiency depends on whether n is product of small primes(ideally power of two)

If not, then much of computational advantage of FFT maybe lost

For example, if n itself is prime, then sequence cannot besplit at all, and “fast” algorithm becomes standard O(n2)matrix-vector multiplication

Michael T. Heath Scientific Computing 22 / 32Cooley-Tukey FFT algorithm

Page 28: Numerical Analysis Fast Fourier Transform

Applications IDiscrete Fourier Transform

Fast Fourier TransformApplications

Applications of DFT

DFT is often of direct interest itself and is also useful ascomputational tool that provides efficient means forcomputing other quantities

By its nature, DFT can be used to detect periodicities orcycles in discrete data, and to remove unwantedperiodicities

For example, to remove high-frequency noise, computeDFT of sequence, set high-frequency components oftransformed sequence to zero, then compute inverse DFTof modified sequence to get back into original domain

Michael T. Heath Scientific Computing 23 / 32

Page 29: Numerical Analysis Fast Fourier Transform

Applications IIDiscrete Fourier Transform

Fast Fourier TransformApplications

Applications of DFT, continued

As another example, weather data often contain twodistinct cycles, diurnal (daily) and annual (yearly), and onemight want to remove one to study the other in isolation

Economic data are also often “seasonally adjusted” byremoving unwanted periodicities to reveal “secular” trends

Because of such uses, DFT is of vital importance in manyaspects of signal processing, such as digital filtering

< interactive example >

Michael T. Heath Scientific Computing 24 / 32

Page 30: Numerical Analysis Fast Fourier Transform

Applications IIIDiscrete Fourier Transform

Fast Fourier TransformApplications

Applications of DFT, continued

Some computations are simpler or more efficient infrequency domain than in time domain

Examples include discrete convolution of two sequences uand v of length n

{u ? v}m =n−1∑

k=0

vkum−k, m = 0, 1, . . . , n− 1

and related quantities such as cross-correlation of twosequences or autocorrelation of a sequence with itself

Equivalent operation in frequency domain is simplypointwise multiplication (plus complex conjugation in somecases)

Michael T. Heath Scientific Computing 25 / 32

Page 31: Numerical Analysis Fast Fourier Transform

Applications IVDiscrete Fourier Transform

Fast Fourier TransformApplications

Applications of DFT, continued

If DFT and its inverse can be computed efficiently, then itmay be advantageous to transform to frequency domain tocompute such convolutions, then transform back to timedomain

For example, to compute convolution or correlation of twosequences, it is often advantageous to use FFT algorithmto compute DFT of both sequences, compute pointwiseproduct in frequency domain, then inverse DFT back totime domain, again via FFT algorithm

FFT algorithm also forms basis for exceptionally efficientmethods for solving certain periodic boundary valueproblems, such as Poisson’s equation on regular domainwith periodic boundary conditions

Michael T. Heath Scientific Computing 26 / 32

Demo

Page 32: Numerical Analysis Fast Fourier Transform

Applications IVDiscrete Fourier Transform

Fast Fourier TransformApplications

Applications of DFT, continued

If DFT and its inverse can be computed efficiently, then itmay be advantageous to transform to frequency domain tocompute such convolutions, then transform back to timedomain

For example, to compute convolution or correlation of twosequences, it is often advantageous to use FFT algorithmto compute DFT of both sequences, compute pointwiseproduct in frequency domain, then inverse DFT back totime domain, again via FFT algorithm

FFT algorithm also forms basis for exceptionally efficientmethods for solving certain periodic boundary valueproblems, such as Poisson’s equation on regular domainwith periodic boundary conditions

Michael T. Heath Scientific Computing 26 / 32Demo

Page 33: Numerical Analysis Fast Fourier Transform

Fast polynomial multiplicationDiscrete Fourier Transform

Fast Fourier TransformApplications

Fast Polynomial Multiplication

FFT algorithm also provides fast methods for somecomputations that might not seem related to it

For example, complexity of straightforward multiplication oftwo polynomials is proportional to product of their degrees

However, polynomial of degree n− 1 is uniquelydetermined by its values at n distinct points

Thus, product polynomial can be determined byinterpolation from pointwise product of factor polynomialsevaluated at n points

Both polynomial evaluation and interpolation using n pointswould normally require O(n2) operations, but by choosingpoints to be nth roots of unity, FFT algorithm can be usedto reduce complexity to O(n log2 n)

Michael T. Heath Scientific Computing 27 / 32

Page 34: Numerical Analysis Fast Fourier Transform

Wavelets IDiscrete Fourier Transform

Fast Fourier TransformApplications

Wavelets

Sine and cosine functions used in Fourier analysis are verysmooth (infinitely differentiable), and very broad (nonzeroalmost everywhere on real line)

They are not very effective for representing functions thatchange abruptly or have highly localized support

Gibbs phenomenon in Fourier representation of squarewave (“ringing” at corners) is one manifestation of this

In response to this shortcoming, there has been intenseinterest in recent years in new type of basis functionscalled wavelets

Michael T. Heath Scientific Computing 28 / 32

Page 35: Numerical Analysis Fast Fourier Transform

Wavelets IIDiscrete Fourier Transform

Fast Fourier TransformApplications

Wavelets, continued

Wavelet basis is generated from single function φ(x),called mother wavelet or scaling function, by dilation andtranslation, φ((x− b)/a), where a, b ∈ R with a 6= 0

There are many choices for mother wavelet, with choicetrading off smoothness vs compactness

Commonly used family of wavelets is due to Daubechies,example of which is shown below

Michael T. Heath Scientific Computing 29 / 32

Page 36: Numerical Analysis Fast Fourier Transform

Wavelets IIIDiscrete Fourier Transform

Fast Fourier TransformApplications

Wavelets, continued

Typical choices for dilation and translation parameters area = 2−j and b = k2j , where j and k are integers, so thatφjk(x) = φ(2jx− k)

If mother wavelet φ(x) has sufficiently localized support,then ∫

φjkφmn = 0

whenever indices do not both match, so doubly-indexedbasis functions φjk(x) are orthogonal

By replicating mother wavelet at many different scales, it ispossible to mimic behavior of any function at many differentscales; this property of wavelets is called multiresolution

Michael T. Heath Scientific Computing 30 / 32

Page 37: Numerical Analysis Fast Fourier Transform

Wavelets IVDiscrete Fourier Transform

Fast Fourier TransformApplications

Wavelets, continued

Fourier basis functions are localized in frequency but not intime: small changes in frequency produce changeseverywhere in time domain

Wavelets are localized in both frequency (by dilation) andtime (by translation)

This localization tends to make wavelet representation offunction very sparse

Michael T. Heath Scientific Computing 31 / 32

Page 38: Numerical Analysis Fast Fourier Transform

DWTDiscrete Fourier Transform

Fast Fourier TransformApplications

Discrete Wavelet Transform

As with Fourier transform, there is analogous discretewavelet transform, or DWT

DWT and its inverse can be computed very efficiently bypyramidal, hierarchical algorithm

Sparsity of wavelet basis makes computation of DWT evenfaster than FFT

DWT requires only O(n) work for sequence of length n,instead of O(n log n)Because of their efficiency, both in computation and incompactness of representation, wavelets are playing anincreasingly important role in many areas of signal andimage processing, such as data compression, noiseremoval, and computer vision

Michael T. Heath Scientific Computing 32 / 32

Page 39: Numerical Analysis Fast Fourier Transform

Questions?

?