Numerical Analysis Fast Fourier Transform

Numerical AnalysisFast Fourier Transform

Andreas Kloeckner 〈andreask@illinois.edu〉Base slides c© Michael T. Heath

CS 450 · December 6, 2013

Goal today:Think about data in the frequency domain.

Audio synthesis demo

What about cosines?

Audio synthesis demo

What about cosines?

Complex Exponential NotationDiscrete Fourier Transform

Fast Fourier TransformApplications

Complex Exponential NotationWe will use complex exponential notation based on Euler’sidentity

eiθ = cos θ + i sin θ

where i =√−1

Since e−iθ = cos(−θ) + i sin(−θ) = cos θ − i sin θ, we have

cos(2πkt) =e2πikt + e−2πikt

sin(2πkt) = ie−2πikt − e2πikt

Pure cosine or sine wave of frequency k is equivalent tosum or difference of complex exponentials of halfamplitude and frequencies k and −k

Michael T. Heath Scientific Computing 4 / 32

Roots of unityDiscrete Fourier Transform

Roots of UnityFor given integer n, primitive nth root of unity is given by

ωn = cos(2π/n)− i sin(2π/n) = e−2πi/n

nth roots of unity, called twiddle factors in this context, aregiven by ωkn or by ω−k

n , k = 0, . . . , n− 1

< interactive example >Michael T. Heath Scientific Computing 5 / 32

Synthesis with roots of unitydemo

Trigonometric InterpolationDiscrete Fourier Transform

Trigonometric Interpolation

In modeling periodic or cyclic phenomena, sines andcosines are more appropriate functions than polynomialsor piecewise polynomials

Representation as linear combination of sines and cosinesdecomposes continuous function or discrete data intocomponents of various frequencies

Representation in frequency space may enable moreefficient manipulations than in original time or spacedomain

Audio demo

Questions

Why is there odd distortion in the audio?What is the cost of applying that matrix?Can we do this more cheaply?Can we do also apply the inverse more cheaply?

DFTDiscrete Fourier Transform

Discrete Fourier Transform

Discrete Fourier transform, or DFT, of sequencex = [x0, . . . , xn−1]

T is sequence y = [y0, . . . , yn−1]T given

ym =n−1∑

xk ωmkn , m = 0, 1, . . . , n− 1

In matrix notation, y = Fn x, where entries of Fouriermatrix Fn are given by

{Fn}mk = ωmkn

For example,

1 1 1 11 ω1 ω2 ω3

1 ω2 ω4 ω6

1 ω3 ω6 ω9

1 1 1 11 −i −1 i1 −1 1 −11 i −1 −i

Inverse DFTDiscrete Fourier Transform

Inverse DFT

Note that

1 1 1 11 ω−1 ω−2 ω−3

1 ω−2 ω−4 ω−6

1 ω−3 ω−6 ω−9

1 1 1 11 ω1 ω2 ω3

1 ω2 ω4 ω6

1 ω3 ω6 ω9

1 0 0 00 1 0 00 0 1 00 0 0 1

In general,F−1n = (1/n)FH

Inverse DFT is therefore given by

n−1∑

ym ω−mkn k = 0, 1, . . . , n− 1

DFT gives trigonometric interpolant using onlymatrix-vector multiplication, which costs only O(n2)

More DFTDiscrete Fourier Transform

DFT, continued

DFT of sequence, even purely real sequence, is in generalcomplex

Components of DFT y of real sequence x of length n areconjugate symmetric : yk and yn−k are complex conjugatesfor k = 1, . . . , (n/2)− 1

Two components of special interest arey0, whose value is sum of components of x, is sometimescalled DC component, corresponding to zero frequency(i.e., constant function)yn/2, corresponding to Nyquist frequency, which is highestfrequency representable at given sampling rate

Components of y beyond Nyquist frequency correspond tofrequencies that are negatives of those below Nyquistfrequency

ExampleDiscrete Fourier Transform

Example: DFT

For cyclic sequence x,

F8 x = F8

1−11−11−11−1

00008000

Sequence has highest possible rate of oscillation (between1 and −1) for this sampling rate

In transformed sequence, only nonzero component is atNyquist frequency (in this case y4)

Computing the DFT IDiscrete Fourier Transform

Computing DFT

By taking advantage of symmetries and redundancies indefinition of DFT, shortcut algorithm can be developed forevaluating DFT very efficiently

For illustration, consider case n = 4

From definition of DFT

ym =3∑

xk ωmkn , m = 0, . . . , 3

Writing out four equations in full

y0 = x0ω0n + x1ω

0n + x2ω

0n + x3ω

y1 = x0ω0n + x1ω

1n + x2ω

2n + x3ω

y2 = x0ω0n + x1ω

2n + x2ω

4n + x3ω

y3 = x0ω0n + x1ω

3n + x2ω

6n + x3ω

Computing the DFT IIDiscrete Fourier Transform

Computing DFT, continuedNoting that

ω0n = ω4

n = 1, ω2n = ω6

n = −1, ω9n = ω1

and regrouping, we obtain four equations

y0 = (x0 + ω0nx2) + ω0

n(x1 + ω0nx3)

y1 = (x0 − ω0nx2) + ω1

n(x1 − ω0nx3)

y2 = (x0 + ω0nx2) + ω2

n(x1 + ω0nx3)

y3 = (x0 − ω0nx2) + ω3

n(x1 − ω0nx3)

DFT can now be computed with only 8 additions orsubtractions and 6 multiplications, instead of expected(4− 1) ∗ 4 = 12 additions and 42 = 16 multiplications

Actually, even fewer multiplications are required for thissmall case, since ω0

n = 1, but we have tried to illustrate howalgorithm works in general

Computing the DFT IIIDiscrete Fourier Transform

Computing DFT, continued

Main point is that computing DFT of original 4-pointsequence has been reduced to computing DFT of its two2-point even and odd subsequences

This property holds in general: DFT of n-point sequencecan be computed by breaking it into two DFTs of halflength, provided n is even

General pattern becomes clearer when viewed in terms offirst few Fourier matrices

F1 = 1, F2 =

[1 11 −1

], F4 =

1 1 1 11 −i −1 i1 −1 1 −11 i −1 −i

, . . .

Computing the DFT IV

Xk =N−1∑

xne− 2πiN nk

N/2−1∑

x2me− 2πiN (2m)k +

N/2−1∑

x2m+1e− 2πiN (2m+1)k

N/2−1∑

x2me− 2πiN/2 mk

︸︷︷︸DFT of even−indexed part of xm

+ e− 2πiN k

N/2−1∑

x2m+1e− 2πiN/2 mk

︸︷︷︸DFT of odd−indexed part of xm

= Ek + e− 2πiN kOk .

Summary: (Power-of-2) algorithmDiscrete Fourier Transform

FFT Algorithm

procedure fft(x, y, n, ω)if n = 1 then

y[0] = x[0]else

for k = 0 to (n/2)− 1p[k] = x[2k]s[k] = x[2k + 1]

endfft(p, q, n/2, ω2)fft(s, t, n/2, ω2)for k = 0 to n− 1

y[k] = q[k mod (n/2)]+ωkt[k mod (n/2)]

endend

{ bottom of recursion }

{ split into even andodd subsequences }

{ call fft procedurerecursively }

{ combine results }

Complexity?

Summary: (Power-of-2) algorithmDiscrete Fourier Transform

FFT Algorithm

procedure fft(x, y, n, ω)if n = 1 then

y[0] = x[0]else

for k = 0 to (n/2)− 1p[k] = x[2k]s[k] = x[2k + 1]

endfft(p, q, n/2, ω2)fft(s, t, n/2, ω2)for k = 0 to n− 1

y[k] = q[k mod (n/2)]+ωkt[k mod (n/2)]

endend

{ bottom of recursion }

{ split into even andodd subsequences }

{ call fft procedurerecursively }

{ combine results }

Michael T. Heath Scientific Computing 16 / 32Complexity?

(Power-of-2) algorithm: Remarks IDiscrete Fourier Transform

FFT Algorithm, continued

There are log2 n levels of recursion, each of which involvesO(n) arithmetic operations, so total cost is O(n log2 n)

For clarity, separate arrays were used for subsequences,but transform can be computed in place using no additionalstorage

Input sequence is assumed complex; if input sequence isreal, then additional symmetries in DFT can be exploited toreduce storage and operation count by half

Output sequence is not produced in natural order, buteither input or output sequence can be rearranged at costof O(n log2 n), analogous to sorting

(Power-of-2) algorithm: Remarks IIDiscrete Fourier Transform

FFT Algorithm, continued

FFT algorithm can be formulated using iteration rather thanrecursion, which is often desirable for greater efficiency orwhen using programming language that does not supportrecursion

Despite its name, fast Fourier transform is an algorithm,not a transform

It is particular way of computing DFT of sequence inefficient manner

< interactive example >

ComplexityDiscrete Fourier Transform

Complexity of FFT

DFT is defined in terms of matrix-vector product, whosestraightforward evaluation would appear to require O(n2)arithmetic operations

Use of FFT algorithm reduces work to only O(n log2 n),which makes enormous practical difference in timerequired to transform large sequences

n n log2 n n2

64 384 4096128 896 16384256 2048 65536512 4608 262144

1024 10240 1048576

Inverse FFTDiscrete Fourier Transform

Inverse Transform

Due to similar form of DFT and its inverse (only sign ofexponent differs), FFT algorithm can also be used tocompute inverse DFT efficiently

Ability to transform back and forth quickly between timeand frequency domains makes it practical to performcomputations or analysis that may be required inwhichever domain is more convenient and efficient

LimitationsDiscrete Fourier Transform

Limitations of FFT

FFT algorithm is not always applicable or maximallyefficientInput sequence is assumed to be

Equally spacedPeriodicPower of two in length

First two of these follow from definition of DFT, while thirdis required for maximal efficiency of FFT algorithm

Care must be taken in applying FFT algorithm to producemost meaningful results as efficiently as possible

For example, transforming sequence that is not reallyperiodic or padding sequence to make its length power oftwo may introduce spurious noise and complicateinterpretation of results < interactive example >

What about n = pq?Discrete Fourier Transform

Mixed-Radix FFT

Mixed-radix FFT algorithm does not require number ofpoints n to be power of two

More general algorithm is still based on divide andconquer, with sequence being split at each level bysmallest prime factor of length of remaining sequence

Efficiency depends on whether n is product of small primes(ideally power of two)

If not, then much of computational advantage of FFT maybe lost

For example, if n itself is prime, then sequence cannot besplit at all, and “fast” algorithm becomes standard O(n2)matrix-vector multiplication

Cooley-Tukey FFT algorithm

What about n = pq?Discrete Fourier Transform

Mixed-Radix FFT

Mixed-radix FFT algorithm does not require number ofpoints n to be power of two

More general algorithm is still based on divide andconquer, with sequence being split at each level bysmallest prime factor of length of remaining sequence

Efficiency depends on whether n is product of small primes(ideally power of two)

If not, then much of computational advantage of FFT maybe lost

For example, if n itself is prime, then sequence cannot besplit at all, and “fast” algorithm becomes standard O(n2)matrix-vector multiplication

Michael T. Heath Scientific Computing 22 / 32Cooley-Tukey FFT algorithm

Applications IDiscrete Fourier Transform

Applications of DFT

DFT is often of direct interest itself and is also useful ascomputational tool that provides efficient means forcomputing other quantities

By its nature, DFT can be used to detect periodicities orcycles in discrete data, and to remove unwantedperiodicities

For example, to remove high-frequency noise, computeDFT of sequence, set high-frequency components oftransformed sequence to zero, then compute inverse DFTof modified sequence to get back into original domain

Applications IIDiscrete Fourier Transform

Applications of DFT, continued

As another example, weather data often contain twodistinct cycles, diurnal (daily) and annual (yearly), and onemight want to remove one to study the other in isolation

Economic data are also often “seasonally adjusted” byremoving unwanted periodicities to reveal “secular” trends

Because of such uses, DFT is of vital importance in manyaspects of signal processing, such as digital filtering

< interactive example >

Applications IIIDiscrete Fourier Transform

Some computations are simpler or more efficient infrequency domain than in time domain

Examples include discrete convolution of two sequences uand v of length n

{u ? v}m =n−1∑

vkum−k, m = 0, 1, . . . , n− 1

and related quantities such as cross-correlation of twosequences or autocorrelation of a sequence with itself

Equivalent operation in frequency domain is simplypointwise multiplication (plus complex conjugation in somecases)

Applications IVDiscrete Fourier Transform

If DFT and its inverse can be computed efficiently, then itmay be advantageous to transform to frequency domain tocompute such convolutions, then transform back to timedomain

For example, to compute convolution or correlation of twosequences, it is often advantageous to use FFT algorithmto compute DFT of both sequences, compute pointwiseproduct in frequency domain, then inverse DFT back totime domain, again via FFT algorithm

FFT algorithm also forms basis for exceptionally efficientmethods for solving certain periodic boundary valueproblems, such as Poisson’s equation on regular domainwith periodic boundary conditions

Applications IVDiscrete Fourier Transform

If DFT and its inverse can be computed efficiently, then itmay be advantageous to transform to frequency domain tocompute such convolutions, then transform back to timedomain

For example, to compute convolution or correlation of twosequences, it is often advantageous to use FFT algorithmto compute DFT of both sequences, compute pointwiseproduct in frequency domain, then inverse DFT back totime domain, again via FFT algorithm

FFT algorithm also forms basis for exceptionally efficientmethods for solving certain periodic boundary valueproblems, such as Poisson’s equation on regular domainwith periodic boundary conditions

Michael T. Heath Scientific Computing 26 / 32Demo

Fast polynomial multiplicationDiscrete Fourier Transform

Fast Polynomial Multiplication

FFT algorithm also provides fast methods for somecomputations that might not seem related to it

For example, complexity of straightforward multiplication oftwo polynomials is proportional to product of their degrees

However, polynomial of degree n− 1 is uniquelydetermined by its values at n distinct points

Thus, product polynomial can be determined byinterpolation from pointwise product of factor polynomialsevaluated at n points

Both polynomial evaluation and interpolation using n pointswould normally require O(n2) operations, but by choosingpoints to be nth roots of unity, FFT algorithm can be usedto reduce complexity to O(n log2 n)

Wavelets IDiscrete Fourier Transform

Wavelets

Sine and cosine functions used in Fourier analysis are verysmooth (infinitely differentiable), and very broad (nonzeroalmost everywhere on real line)

They are not very effective for representing functions thatchange abruptly or have highly localized support

Gibbs phenomenon in Fourier representation of squarewave (“ringing” at corners) is one manifestation of this

In response to this shortcoming, there has been intenseinterest in recent years in new type of basis functionscalled wavelets

Wavelets IIDiscrete Fourier Transform

Wavelets, continued

Wavelet basis is generated from single function φ(x),called mother wavelet or scaling function, by dilation andtranslation, φ((x− b)/a), where a, b ∈ R with a 6= 0

There are many choices for mother wavelet, with choicetrading off smoothness vs compactness

Commonly used family of wavelets is due to Daubechies,example of which is shown below

Wavelets IIIDiscrete Fourier Transform

Wavelets, continued

Typical choices for dilation and translation parameters area = 2−j and b = k2j , where j and k are integers, so thatφjk(x) = φ(2jx− k)

If mother wavelet φ(x) has sufficiently localized support,then ∫

φjkφmn = 0

whenever indices do not both match, so doubly-indexedbasis functions φjk(x) are orthogonal

By replicating mother wavelet at many different scales, it ispossible to mimic behavior of any function at many differentscales; this property of wavelets is called multiresolution

Wavelets IVDiscrete Fourier Transform

Wavelets, continued

Fourier basis functions are localized in frequency but not intime: small changes in frequency produce changeseverywhere in time domain

Wavelets are localized in both frequency (by dilation) andtime (by translation)

This localization tends to make wavelet representation offunction very sparse

DWTDiscrete Fourier Transform

Discrete Wavelet Transform

As with Fourier transform, there is analogous discretewavelet transform, or DWT

DWT and its inverse can be computed very efficiently bypyramidal, hierarchical algorithm

Sparsity of wavelet basis makes computation of DWT evenfaster than FFT

DWT requires only O(n) work for sequence of length n,instead of O(n log n)Because of their efficiency, both in computation and incompactness of representation, wavelets are playing anincreasingly important role in many areas of signal andimage processing, such as data compression, noiseremoval, and computer vision

Questions?

Numerical Analysis Fast Fourier Transform

Documents

Transcript of Numerical Analysis Fast Fourier Transform

The Discrete Cosine Transform

Image Processing Image Transform and Fourier/Wavelet Transform

Pressure laws and fast Legendre transform - Philippe LeFloch

Περιοδικό transform! 4

Fast and accurate techniques for computing Schur ... · Fast and accurate techniques for computing Schur complements and performing numerical coarse graining Gunnar Martinsson The

Laplace transform

The Partial Fast Fourier Transformbowman/publications/partialfft.pdf · Fourier transform is achieved, with a lower overall coe cient than obtained by Ying and Fomel. Keywords partial

Fast direct solvers - Applied Mathematics...Discretization of integral operators typically results in dense matrices. In the 1950’s when computers made numerical PDE solvers possible,

Numerical Analysis - Part II...Alternatively, we can transform A to upper Hessenberg using Householder re ections, rather than Givens rotations. In that case we deal with a column

Numerical Dynamic Programming

Discrete-Time Fourier Transform Discrete Fourier Transform z-Transform

Fourier transform

Unequally spaced FFT and fast Radon transform

Discrete Fourier Transform & Fast Fourier Transformperrins/class/F14_360/lab/labnotes9.pdf · Discrete Fourier Transform (DFT) DFT is the workhorse for Fourier Analysis in MATLAB!

Discrete-time windowing Discrete Fourier Transform ...web.cecs.pdx.edu/~ece2xx/ECE223/Slides/FFT.pdf · Fast Fourier Transform • Discrete-time windowing • Discrete Fourier Transform

Fast and accurate computation of the Euclidean distance transform ...

The z Transform - UPTshannon.etc.upt.ro/teaching/sp-pi/Course/4_z_Transform_2013.pdf · 1 1 The z Transform The z transform generalizes the Discrete-time Fourier Transform for the

Fourier series of periodic discrete-time signals ALGORITHM FOR SOLUTION: Fast Fourier Transform (FFT) - FFT: N log2 N complex additions and (N/2)log2 N complex multiplications (compare

Fast Computation of Fourier Integral Operatorsstatweb.stanford.edu/~candes/papers/FastFIO.pdfFourier transform of f deﬁned by fˆ(ξ) = Z Rd f(x)e−2πixξ dx. (1.2) With the proper

FAST FORWARD FESTIVAL