of 460

• date post

25-Jul-2018
• Category

## Documents

• view

243

3

Embed Size (px)

### Transcript of An Introduction to Statistical Signal Processing - .An Introduction to Statistical Signal Processing

• An Introduction toStatistical Signal Processing

Pr(f F ) = P ({ : F}) = P (f1(F ))

f1(F )f

F

May 5, 2000

• ii

• An Introduction toStatistical Signal Processing

Robert M. Grayand

Lee D. Davisson

Information Systems LaboratoryDepartment of Electrical Engineering

Stanford Universityand

Department of Electrical Engineering and Computer ScienceUniversity of Maryland

• iv

c1999 by the authors.

• v

to our Families

• vi

• Contents

Preface xi

Glossary xv

1 Introduction 1

2 Probability 112.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Spinning Pointers and Flipping Coins . . . . . . . . . . . . 152.3 Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . 23

2.3.1 Sample Spaces . . . . . . . . . . . . . . . . . . . . . 282.3.2 Event Spaces . . . . . . . . . . . . . . . . . . . . . . 312.3.3 Probability Measures . . . . . . . . . . . . . . . . . . 42

2.4 Discrete Probability Spaces . . . . . . . . . . . . . . . . . . 452.5 Continuous Probability Spaces . . . . . . . . . . . . . . . . 562.6 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . 702.7 Elementary Conditional Probability . . . . . . . . . . . . . 712.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

3 Random Objects 853.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

3.1.1 Random Variables . . . . . . . . . . . . . . . . . . . 853.1.2 Random Vectors . . . . . . . . . . . . . . . . . . . . 893.1.3 Random Processes . . . . . . . . . . . . . . . . . . . 93

3.2 Random Variables . . . . . . . . . . . . . . . . . . . . . . . 953.3 Distributions of Random Variables . . . . . . . . . . . . . . 104

3.3.1 Distributions . . . . . . . . . . . . . . . . . . . . . . 1043.3.2 Mixture Distributions . . . . . . . . . . . . . . . . . 1083.3.3 Derived Distributions . . . . . . . . . . . . . . . . . 111

3.4 Random Vectors and Random Processes . . . . . . . . . . . 1153.5 Distributions of Random Vectors . . . . . . . . . . . . . . . 117

vii

• viii CONTENTS

3.5.1 Multidimensional Events . . . . . . . . . . . . . . . 1183.5.2 Multidimensional Probability Functions . . . . . . . 1193.5.3 Consistency of Joint and Marginal Distributions . . 120

3.6 Independent Random Variables . . . . . . . . . . . . . . . . 1273.6.1 IID Random Vectors . . . . . . . . . . . . . . . . . . 128

3.7 Conditional Distributions . . . . . . . . . . . . . . . . . . . 1293.7.1 Discrete Conditional Distributions . . . . . . . . . . 1303.7.2 Continuous Conditional Distributions . . . . . . . . 131

3.8 Statistical Detection and Classification . . . . . . . . . . . . 1343.9 Additive Noise . . . . . . . . . . . . . . . . . . . . . . . . . 1373.10 Binary Detection in Gaussian Noise . . . . . . . . . . . . . 1443.11 Statistical Estimation . . . . . . . . . . . . . . . . . . . . . 1463.12 Characteristic Functions . . . . . . . . . . . . . . . . . . . . 1473.13 Gaussian Random Vectors . . . . . . . . . . . . . . . . . . . 1523.14 Examples: Simple Random Processes . . . . . . . . . . . . . 1543.15 Directly Given Random Processes . . . . . . . . . . . . . . 157

3.15.1 The Kolmogorov Extension Theorem . . . . . . . . . 1573.15.2 IID Random Processes . . . . . . . . . . . . . . . . . 1583.15.3 Gaussian Random Processes . . . . . . . . . . . . . . 158

3.16 Discrete Time Markov Processes . . . . . . . . . . . . . . . 1593.16.1 A Binary Markov Process . . . . . . . . . . . . . . . 1593.16.2 The Binomial Counting Process . . . . . . . . . . . . 1623.16.3 Discrete Random Walk . . . . . . . . . . . . . . . . 1653.16.4 The Discrete Time Wiener Process . . . . . . . . . . 1663.16.5 Hidden Markov Models . . . . . . . . . . . . . . . . 167

3.17 Nonelementary Conditional Probability . . . . . . . . . . . 1683.18 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

4 Expectation and Averages 1874.1 Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1874.2 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

4.2.1 Examples: Expectation . . . . . . . . . . . . . . . . 1924.3 Functions of Several Random Variables . . . . . . . . . . . . 2004.4 Properties of Expectation . . . . . . . . . . . . . . . . . . . 2004.5 Examples: Functions of Several Random Variables . . . . . 203

4.5.1 Correlation . . . . . . . . . . . . . . . . . . . . . . . 2034.5.2 Covariance . . . . . . . . . . . . . . . . . . . . . . . 2054.5.3 Covariance Matrices . . . . . . . . . . . . . . . . . . 2064.5.4 Multivariable Characteristic Functions . . . . . . . . 2074.5.5 Example: Differential Entropy of a Gaussian Vector 209

4.6 Conditional Expectation . . . . . . . . . . . . . . . . . . . . 2104.7 Jointly Gaussian Vectors . . . . . . . . . . . . . . . . . . . 213

• CONTENTS ix

4.8 Expectation as Estimation . . . . . . . . . . . . . . . . . . . 2164.9 Implications for Linear Estimation . . . . . . . . . . . . . 2224.10 Correlation and Linear Estimation . . . . . . . . . . . . . . 2244.11 Correlation and Covariance Functions . . . . . . . . . . . . 2314.12 The Central Limit Theorem . . . . . . . . . . . . . . . . . 2354.13 Sample Averages . . . . . . . . . . . . . . . . . . . . . . . . 2374.14 Convergence of Random Variables . . . . . . . . . . . . . . 2394.15 Weak Law of Large Numbers . . . . . . . . . . . . . . . . . 2444.16 Strong Law of Large Numbers . . . . . . . . . . . . . . . . 2464.17 Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . . 2514.18 Asymptotically Uncorrelated Processes . . . . . . . . . . . . 2564.19 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

5 Second-Order Moments 2815.1 Linear Filtering of Random Processes . . . . . . . . . . . . 2825.2 Second-Order Linear Systems I/O Relations . . . . . . . . . 2845.3 Power Spectral Densities . . . . . . . . . . . . . . . . . . . . 2895.4 Linearly Filtered Uncorrelated Processes . . . . . . . . . . . 2925.5 Linear Modulation . . . . . . . . . . . . . . . . . . . . . . . 2985.6 White Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . 3015.7 Time-Averages . . . . . . . . . . . . . . . . . . . . . . . . . 3055.8 Differentiating Random Processes . . . . . . . . . . . . . . 3095.9 Linear Estimation and Filtering . . . . . . . . . . . . . . . 3125.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326

6 A Menagerie of Processes 3436.1 Discrete Time Linear Models . . . . . . . . . . . . . . . . . 3446.2 Sums of IID Random Variables . . . . . . . . . . . . . . . . 3486.3 Independent Stationary Increments . . . . . . . . . . . . . . 3506.4 Second-Order Moments of ISI Processes . . . . . . . . . . 3536.5 Specification of Continuous Time ISI Processes . . . . . . . 3556.6 Moving-Average and Autoregressive Processes . . . . . . . . 3586.7 The Discrete Time Gauss-Markov Process . . . . . . . . . . 3606.8 Gaussian Random Processes . . . . . . . . . . . . . . . . . . 3616.9 The Poisson Counting Process . . . . . . . . . . . . . . . . 3616.10 Compound Processes . . . . . . . . . . . . . . . . . . . . . . 3646.11 Exponential Modulation . . . . . . . . . . . . . . . . . . . 3666.12 Thermal Noise . . . . . . . . . . . . . . . . . . . . . . . . . 3716.13 Ergodicity and Strong Laws of Large Numbers . . . . . . . 3736.14 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

• x CONTENTS

A Preliminaries 389A.1 Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 389A.2 Examples of Proofs . . . . . . . . . . . . . . . . . . . . . . . 397A.3 Mappings and Functions . . . . . . . . . . . . . . . . . . . . 401A.4 Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . 402A.5 Linear System Fundamentals . . . . . . . . . . . . . . . . . 405A.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410

B Sums and Integrals 417B.1 Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . 417B.2 Double Sums . . . . . . . . . . . . . . . . . . . . . . . . . . 420B.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . 421B.4 The Lebesgue Integral . . . . . . . . . . . . . . . . . . . . 423

C Common Univariate Distributions 427