Numerical AnalysisFast Fourier Transform
Andreas Kloeckner 〈[email protected]〉Base slides c© Michael T. Heath
CS 450 · December 6, 2013
Goal today:Think about data in the frequency domain.
Audio synthesis demo
What about cosines?
Audio synthesis demo
What about cosines?
Complex Exponential NotationDiscrete Fourier Transform
Fast Fourier TransformApplications
Complex Exponential NotationWe will use complex exponential notation based on Euler’sidentity
eiθ = cos θ + i sin θ
where i =√−1
Since e−iθ = cos(−θ) + i sin(−θ) = cos θ − i sin θ, we have
cos(2πkt) =e2πikt + e−2πikt
2
and
sin(2πkt) = ie−2πikt − e2πikt
2
Pure cosine or sine wave of frequency k is equivalent tosum or difference of complex exponentials of halfamplitude and frequencies k and −k
Michael T. Heath Scientific Computing 4 / 32
Roots of unityDiscrete Fourier Transform
Fast Fourier TransformApplications
Roots of UnityFor given integer n, primitive nth root of unity is given by
ωn = cos(2π/n)− i sin(2π/n) = e−2πi/n
nth roots of unity, called twiddle factors in this context, aregiven by ωkn or by ω−k
n , k = 0, . . . , n− 1
< interactive example >Michael T. Heath Scientific Computing 5 / 32
Synthesis with roots of unitydemo
Trigonometric InterpolationDiscrete Fourier Transform
Fast Fourier TransformApplications
Trigonometric Interpolation
In modeling periodic or cyclic phenomena, sines andcosines are more appropriate functions than polynomialsor piecewise polynomials
Representation as linear combination of sines and cosinesdecomposes continuous function or discrete data intocomponents of various frequencies
Representation in frequency space may enable moreefficient manipulations than in original time or spacedomain
Michael T. Heath Scientific Computing 3 / 32
Audio demo
Questions
Why is there odd distortion in the audio?What is the cost of applying that matrix?Can we do this more cheaply?Can we do also apply the inverse more cheaply?
DFTDiscrete Fourier Transform
Fast Fourier TransformApplications
Discrete Fourier Transform
Discrete Fourier transform, or DFT, of sequencex = [x0, . . . , xn−1]
T is sequence y = [y0, . . . , yn−1]T given
by
ym =n−1∑
k=0
xk ωmkn , m = 0, 1, . . . , n− 1
In matrix notation, y = Fn x, where entries of Fouriermatrix Fn are given by
{Fn}mk = ωmkn
For example,
F4 =
1 1 1 11 ω1 ω2 ω3
1 ω2 ω4 ω6
1 ω3 ω6 ω9
=
1 1 1 11 −i −1 i1 −1 1 −11 i −1 −i
Michael T. Heath Scientific Computing 6 / 32
Inverse DFTDiscrete Fourier Transform
Fast Fourier TransformApplications
Inverse DFT
Note that
1
n
1 1 1 11 ω−1 ω−2 ω−3
1 ω−2 ω−4 ω−6
1 ω−3 ω−6 ω−9
1 1 1 11 ω1 ω2 ω3
1 ω2 ω4 ω6
1 ω3 ω6 ω9
=
1 0 0 00 1 0 00 0 1 00 0 0 1
In general,F−1n = (1/n)FH
n
Inverse DFT is therefore given by
xk =1
n
n−1∑
m=0
ym ω−mkn k = 0, 1, . . . , n− 1
DFT gives trigonometric interpolant using onlymatrix-vector multiplication, which costs only O(n2)
Michael T. Heath Scientific Computing 7 / 32
More DFTDiscrete Fourier Transform
Fast Fourier TransformApplications
DFT, continued
DFT of sequence, even purely real sequence, is in generalcomplex
Components of DFT y of real sequence x of length n areconjugate symmetric : yk and yn−k are complex conjugatesfor k = 1, . . . , (n/2)− 1
Two components of special interest arey0, whose value is sum of components of x, is sometimescalled DC component, corresponding to zero frequency(i.e., constant function)yn/2, corresponding to Nyquist frequency, which is highestfrequency representable at given sampling rate
Components of y beyond Nyquist frequency correspond tofrequencies that are negatives of those below Nyquistfrequency
Michael T. Heath Scientific Computing 8 / 32
ExampleDiscrete Fourier Transform
Fast Fourier TransformApplications
Example: DFT
For cyclic sequence x,
F8 x = F8
1−11−11−11−1
=
00008000
= y
Sequence has highest possible rate of oscillation (between1 and −1) for this sampling rate
In transformed sequence, only nonzero component is atNyquist frequency (in this case y4)
Michael T. Heath Scientific Computing 10 / 32
Computing the DFT IDiscrete Fourier Transform
Fast Fourier TransformApplications
Computing DFT
By taking advantage of symmetries and redundancies indefinition of DFT, shortcut algorithm can be developed forevaluating DFT very efficiently
For illustration, consider case n = 4
From definition of DFT
ym =3∑
k=0
xk ωmkn , m = 0, . . . , 3
Writing out four equations in full
y0 = x0ω0n + x1ω
0n + x2ω
0n + x3ω
0n
y1 = x0ω0n + x1ω
1n + x2ω
2n + x3ω
3n
y2 = x0ω0n + x1ω
2n + x2ω
4n + x3ω
6n
y3 = x0ω0n + x1ω
3n + x2ω
6n + x3ω
9n
Michael T. Heath Scientific Computing 11 / 32
Computing the DFT IIDiscrete Fourier Transform
Fast Fourier TransformApplications
Computing DFT, continuedNoting that
ω0n = ω4
n = 1, ω2n = ω6
n = −1, ω9n = ω1
n
and regrouping, we obtain four equations
y0 = (x0 + ω0nx2) + ω0
n(x1 + ω0nx3)
y1 = (x0 − ω0nx2) + ω1
n(x1 − ω0nx3)
y2 = (x0 + ω0nx2) + ω2
n(x1 + ω0nx3)
y3 = (x0 − ω0nx2) + ω3
n(x1 − ω0nx3)
DFT can now be computed with only 8 additions orsubtractions and 6 multiplications, instead of expected(4− 1) ∗ 4 = 12 additions and 42 = 16 multiplications
Actually, even fewer multiplications are required for thissmall case, since ω0
n = 1, but we have tried to illustrate howalgorithm works in general
Michael T. Heath Scientific Computing 12 / 32
Computing the DFT IIIDiscrete Fourier Transform
Fast Fourier TransformApplications
Computing DFT, continued
Main point is that computing DFT of original 4-pointsequence has been reduced to computing DFT of its two2-point even and odd subsequences
This property holds in general: DFT of n-point sequencecan be computed by breaking it into two DFTs of halflength, provided n is even
General pattern becomes clearer when viewed in terms offirst few Fourier matrices
F1 = 1, F2 =
[1 11 −1
], F4 =
1 1 1 11 −i −1 i1 −1 1 −11 i −1 −i
, . . .
Michael T. Heath Scientific Computing 13 / 32
Computing the DFT IV
Xk =N−1∑
n=0
xne− 2πiN nk
Xk =
N/2−1∑
m=0
x2me− 2πiN (2m)k +
N/2−1∑
m=0
x2m+1e− 2πiN (2m+1)k
Xk =
N/2−1∑
m=0
x2me− 2πiN/2 mk
︸ ︷︷ ︸DFT of even−indexed part of xm
+ e− 2πiN k
N/2−1∑
m=0
x2m+1e− 2πiN/2 mk
︸ ︷︷ ︸DFT of odd−indexed part of xm
= Ek + e− 2πiN kOk .
Summary: (Power-of-2) algorithmDiscrete Fourier Transform
Fast Fourier TransformApplications
FFT Algorithm
procedure fft(x, y, n, ω)if n = 1 then
y[0] = x[0]else
for k = 0 to (n/2)− 1p[k] = x[2k]s[k] = x[2k + 1]
endfft(p, q, n/2, ω2)fft(s, t, n/2, ω2)for k = 0 to n− 1
y[k] = q[k mod (n/2)]+ωkt[k mod (n/2)]
endend
{ bottom of recursion }
{ split into even andodd subsequences }
{ call fft procedurerecursively }
{ combine results }
Michael T. Heath Scientific Computing 16 / 32
Complexity?
Summary: (Power-of-2) algorithmDiscrete Fourier Transform
Fast Fourier TransformApplications
FFT Algorithm
procedure fft(x, y, n, ω)if n = 1 then
y[0] = x[0]else
for k = 0 to (n/2)− 1p[k] = x[2k]s[k] = x[2k + 1]
endfft(p, q, n/2, ω2)fft(s, t, n/2, ω2)for k = 0 to n− 1
y[k] = q[k mod (n/2)]+ωkt[k mod (n/2)]
endend
{ bottom of recursion }
{ split into even andodd subsequences }
{ call fft procedurerecursively }
{ combine results }
Michael T. Heath Scientific Computing 16 / 32Complexity?
(Power-of-2) algorithm: Remarks IDiscrete Fourier Transform
Fast Fourier TransformApplications
FFT Algorithm, continued
There are log2 n levels of recursion, each of which involvesO(n) arithmetic operations, so total cost is O(n log2 n)
For clarity, separate arrays were used for subsequences,but transform can be computed in place using no additionalstorage
Input sequence is assumed complex; if input sequence isreal, then additional symmetries in DFT can be exploited toreduce storage and operation count by half
Output sequence is not produced in natural order, buteither input or output sequence can be rearranged at costof O(n log2 n), analogous to sorting
Michael T. Heath Scientific Computing 17 / 32
(Power-of-2) algorithm: Remarks IIDiscrete Fourier Transform
Fast Fourier TransformApplications
FFT Algorithm, continued
FFT algorithm can be formulated using iteration rather thanrecursion, which is often desirable for greater efficiency orwhen using programming language that does not supportrecursion
Despite its name, fast Fourier transform is an algorithm,not a transform
It is particular way of computing DFT of sequence inefficient manner
< interactive example >
Michael T. Heath Scientific Computing 18 / 32
ComplexityDiscrete Fourier Transform
Fast Fourier TransformApplications
Complexity of FFT
DFT is defined in terms of matrix-vector product, whosestraightforward evaluation would appear to require O(n2)arithmetic operations
Use of FFT algorithm reduces work to only O(n log2 n),which makes enormous practical difference in timerequired to transform large sequences
n n log2 n n2
64 384 4096128 896 16384256 2048 65536512 4608 262144
1024 10240 1048576
Michael T. Heath Scientific Computing 19 / 32
Inverse FFTDiscrete Fourier Transform
Fast Fourier TransformApplications
Inverse Transform
Due to similar form of DFT and its inverse (only sign ofexponent differs), FFT algorithm can also be used tocompute inverse DFT efficiently
Ability to transform back and forth quickly between timeand frequency domains makes it practical to performcomputations or analysis that may be required inwhichever domain is more convenient and efficient
Michael T. Heath Scientific Computing 20 / 32
LimitationsDiscrete Fourier Transform
Fast Fourier TransformApplications
Limitations of FFT
FFT algorithm is not always applicable or maximallyefficientInput sequence is assumed to be
Equally spacedPeriodicPower of two in length
First two of these follow from definition of DFT, while thirdis required for maximal efficiency of FFT algorithm
Care must be taken in applying FFT algorithm to producemost meaningful results as efficiently as possible
For example, transforming sequence that is not reallyperiodic or padding sequence to make its length power oftwo may introduce spurious noise and complicateinterpretation of results < interactive example >
Michael T. Heath Scientific Computing 21 / 32
What about n = pq?Discrete Fourier Transform
Fast Fourier TransformApplications
Mixed-Radix FFT
Mixed-radix FFT algorithm does not require number ofpoints n to be power of two
More general algorithm is still based on divide andconquer, with sequence being split at each level bysmallest prime factor of length of remaining sequence
Efficiency depends on whether n is product of small primes(ideally power of two)
If not, then much of computational advantage of FFT maybe lost
For example, if n itself is prime, then sequence cannot besplit at all, and “fast” algorithm becomes standard O(n2)matrix-vector multiplication
Michael T. Heath Scientific Computing 22 / 32
Cooley-Tukey FFT algorithm
What about n = pq?Discrete Fourier Transform
Fast Fourier TransformApplications
Mixed-Radix FFT
Mixed-radix FFT algorithm does not require number ofpoints n to be power of two
More general algorithm is still based on divide andconquer, with sequence being split at each level bysmallest prime factor of length of remaining sequence
Efficiency depends on whether n is product of small primes(ideally power of two)
If not, then much of computational advantage of FFT maybe lost
For example, if n itself is prime, then sequence cannot besplit at all, and “fast” algorithm becomes standard O(n2)matrix-vector multiplication
Michael T. Heath Scientific Computing 22 / 32Cooley-Tukey FFT algorithm
Applications IDiscrete Fourier Transform
Fast Fourier TransformApplications
Applications of DFT
DFT is often of direct interest itself and is also useful ascomputational tool that provides efficient means forcomputing other quantities
By its nature, DFT can be used to detect periodicities orcycles in discrete data, and to remove unwantedperiodicities
For example, to remove high-frequency noise, computeDFT of sequence, set high-frequency components oftransformed sequence to zero, then compute inverse DFTof modified sequence to get back into original domain
Michael T. Heath Scientific Computing 23 / 32
Applications IIDiscrete Fourier Transform
Fast Fourier TransformApplications
Applications of DFT, continued
As another example, weather data often contain twodistinct cycles, diurnal (daily) and annual (yearly), and onemight want to remove one to study the other in isolation
Economic data are also often “seasonally adjusted” byremoving unwanted periodicities to reveal “secular” trends
Because of such uses, DFT is of vital importance in manyaspects of signal processing, such as digital filtering
< interactive example >
Michael T. Heath Scientific Computing 24 / 32
Applications IIIDiscrete Fourier Transform
Fast Fourier TransformApplications
Applications of DFT, continued
Some computations are simpler or more efficient infrequency domain than in time domain
Examples include discrete convolution of two sequences uand v of length n
{u ? v}m =n−1∑
k=0
vkum−k, m = 0, 1, . . . , n− 1
and related quantities such as cross-correlation of twosequences or autocorrelation of a sequence with itself
Equivalent operation in frequency domain is simplypointwise multiplication (plus complex conjugation in somecases)
Michael T. Heath Scientific Computing 25 / 32
Applications IVDiscrete Fourier Transform
Fast Fourier TransformApplications
Applications of DFT, continued
If DFT and its inverse can be computed efficiently, then itmay be advantageous to transform to frequency domain tocompute such convolutions, then transform back to timedomain
For example, to compute convolution or correlation of twosequences, it is often advantageous to use FFT algorithmto compute DFT of both sequences, compute pointwiseproduct in frequency domain, then inverse DFT back totime domain, again via FFT algorithm
FFT algorithm also forms basis for exceptionally efficientmethods for solving certain periodic boundary valueproblems, such as Poisson’s equation on regular domainwith periodic boundary conditions
Michael T. Heath Scientific Computing 26 / 32
Demo
Applications IVDiscrete Fourier Transform
Fast Fourier TransformApplications
Applications of DFT, continued
If DFT and its inverse can be computed efficiently, then itmay be advantageous to transform to frequency domain tocompute such convolutions, then transform back to timedomain
For example, to compute convolution or correlation of twosequences, it is often advantageous to use FFT algorithmto compute DFT of both sequences, compute pointwiseproduct in frequency domain, then inverse DFT back totime domain, again via FFT algorithm
FFT algorithm also forms basis for exceptionally efficientmethods for solving certain periodic boundary valueproblems, such as Poisson’s equation on regular domainwith periodic boundary conditions
Michael T. Heath Scientific Computing 26 / 32Demo
Fast polynomial multiplicationDiscrete Fourier Transform
Fast Fourier TransformApplications
Fast Polynomial Multiplication
FFT algorithm also provides fast methods for somecomputations that might not seem related to it
For example, complexity of straightforward multiplication oftwo polynomials is proportional to product of their degrees
However, polynomial of degree n− 1 is uniquelydetermined by its values at n distinct points
Thus, product polynomial can be determined byinterpolation from pointwise product of factor polynomialsevaluated at n points
Both polynomial evaluation and interpolation using n pointswould normally require O(n2) operations, but by choosingpoints to be nth roots of unity, FFT algorithm can be usedto reduce complexity to O(n log2 n)
Michael T. Heath Scientific Computing 27 / 32
Wavelets IDiscrete Fourier Transform
Fast Fourier TransformApplications
Wavelets
Sine and cosine functions used in Fourier analysis are verysmooth (infinitely differentiable), and very broad (nonzeroalmost everywhere on real line)
They are not very effective for representing functions thatchange abruptly or have highly localized support
Gibbs phenomenon in Fourier representation of squarewave (“ringing” at corners) is one manifestation of this
In response to this shortcoming, there has been intenseinterest in recent years in new type of basis functionscalled wavelets
Michael T. Heath Scientific Computing 28 / 32
Wavelets IIDiscrete Fourier Transform
Fast Fourier TransformApplications
Wavelets, continued
Wavelet basis is generated from single function φ(x),called mother wavelet or scaling function, by dilation andtranslation, φ((x− b)/a), where a, b ∈ R with a 6= 0
There are many choices for mother wavelet, with choicetrading off smoothness vs compactness
Commonly used family of wavelets is due to Daubechies,example of which is shown below
Michael T. Heath Scientific Computing 29 / 32
Wavelets IIIDiscrete Fourier Transform
Fast Fourier TransformApplications
Wavelets, continued
Typical choices for dilation and translation parameters area = 2−j and b = k2j , where j and k are integers, so thatφjk(x) = φ(2jx− k)
If mother wavelet φ(x) has sufficiently localized support,then ∫
φjkφmn = 0
whenever indices do not both match, so doubly-indexedbasis functions φjk(x) are orthogonal
By replicating mother wavelet at many different scales, it ispossible to mimic behavior of any function at many differentscales; this property of wavelets is called multiresolution
Michael T. Heath Scientific Computing 30 / 32
Wavelets IVDiscrete Fourier Transform
Fast Fourier TransformApplications
Wavelets, continued
Fourier basis functions are localized in frequency but not intime: small changes in frequency produce changeseverywhere in time domain
Wavelets are localized in both frequency (by dilation) andtime (by translation)
This localization tends to make wavelet representation offunction very sparse
Michael T. Heath Scientific Computing 31 / 32
DWTDiscrete Fourier Transform
Fast Fourier TransformApplications
Discrete Wavelet Transform
As with Fourier transform, there is analogous discretewavelet transform, or DWT
DWT and its inverse can be computed very efficiently bypyramidal, hierarchical algorithm
Sparsity of wavelet basis makes computation of DWT evenfaster than FFT
DWT requires only O(n) work for sequence of length n,instead of O(n log n)Because of their efficiency, both in computation and incompactness of representation, wavelets are playing anincreasingly important role in many areas of signal andimage processing, such as data compression, noiseremoval, and computer vision
Michael T. Heath Scientific Computing 32 / 32
Questions?
?
Top Related