An Introduction to Statistical Signal Processing - .An Introduction to Statistical Signal Processing

download An Introduction to Statistical Signal Processing - .An Introduction to Statistical Signal Processing

of 460

  • date post

    25-Jul-2018
  • Category

    Documents

  • view

    243
  • download

    3

Embed Size (px)

Transcript of An Introduction to Statistical Signal Processing - .An Introduction to Statistical Signal Processing

  • An Introduction toStatistical Signal Processing

    Pr(f F ) = P ({ : F}) = P (f1(F ))

    f1(F )f

    F

    May 5, 2000

  • ii

  • An Introduction toStatistical Signal Processing

    Robert M. Grayand

    Lee D. Davisson

    Information Systems LaboratoryDepartment of Electrical Engineering

    Stanford Universityand

    Department of Electrical Engineering and Computer ScienceUniversity of Maryland

  • iv

    c1999 by the authors.

  • v

    to our Families

  • vi

  • Contents

    Preface xi

    Glossary xv

    1 Introduction 1

    2 Probability 112.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Spinning Pointers and Flipping Coins . . . . . . . . . . . . 152.3 Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . 23

    2.3.1 Sample Spaces . . . . . . . . . . . . . . . . . . . . . 282.3.2 Event Spaces . . . . . . . . . . . . . . . . . . . . . . 312.3.3 Probability Measures . . . . . . . . . . . . . . . . . . 42

    2.4 Discrete Probability Spaces . . . . . . . . . . . . . . . . . . 452.5 Continuous Probability Spaces . . . . . . . . . . . . . . . . 562.6 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . 702.7 Elementary Conditional Probability . . . . . . . . . . . . . 712.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

    3 Random Objects 853.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

    3.1.1 Random Variables . . . . . . . . . . . . . . . . . . . 853.1.2 Random Vectors . . . . . . . . . . . . . . . . . . . . 893.1.3 Random Processes . . . . . . . . . . . . . . . . . . . 93

    3.2 Random Variables . . . . . . . . . . . . . . . . . . . . . . . 953.3 Distributions of Random Variables . . . . . . . . . . . . . . 104

    3.3.1 Distributions . . . . . . . . . . . . . . . . . . . . . . 1043.3.2 Mixture Distributions . . . . . . . . . . . . . . . . . 1083.3.3 Derived Distributions . . . . . . . . . . . . . . . . . 111

    3.4 Random Vectors and Random Processes . . . . . . . . . . . 1153.5 Distributions of Random Vectors . . . . . . . . . . . . . . . 117

    vii

  • viii CONTENTS

    3.5.1 Multidimensional Events . . . . . . . . . . . . . . . 1183.5.2 Multidimensional Probability Functions . . . . . . . 1193.5.3 Consistency of Joint and Marginal Distributions . . 120

    3.6 Independent Random Variables . . . . . . . . . . . . . . . . 1273.6.1 IID Random Vectors . . . . . . . . . . . . . . . . . . 128

    3.7 Conditional Distributions . . . . . . . . . . . . . . . . . . . 1293.7.1 Discrete Conditional Distributions . . . . . . . . . . 1303.7.2 Continuous Conditional Distributions . . . . . . . . 131

    3.8 Statistical Detection and Classification . . . . . . . . . . . . 1343.9 Additive Noise . . . . . . . . . . . . . . . . . . . . . . . . . 1373.10 Binary Detection in Gaussian Noise . . . . . . . . . . . . . 1443.11 Statistical Estimation . . . . . . . . . . . . . . . . . . . . . 1463.12 Characteristic Functions . . . . . . . . . . . . . . . . . . . . 1473.13 Gaussian Random Vectors . . . . . . . . . . . . . . . . . . . 1523.14 Examples: Simple Random Processes . . . . . . . . . . . . . 1543.15 Directly Given Random Processes . . . . . . . . . . . . . . 157

    3.15.1 The Kolmogorov Extension Theorem . . . . . . . . . 1573.15.2 IID Random Processes . . . . . . . . . . . . . . . . . 1583.15.3 Gaussian Random Processes . . . . . . . . . . . . . . 158

    3.16 Discrete Time Markov Processes . . . . . . . . . . . . . . . 1593.16.1 A Binary Markov Process . . . . . . . . . . . . . . . 1593.16.2 The Binomial Counting Process . . . . . . . . . . . . 1623.16.3 Discrete Random Walk . . . . . . . . . . . . . . . . 1653.16.4 The Discrete Time Wiener Process . . . . . . . . . . 1663.16.5 Hidden Markov Models . . . . . . . . . . . . . . . . 167

    3.17 Nonelementary Conditional Probability . . . . . . . . . . . 1683.18 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

    4 Expectation and Averages 1874.1 Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1874.2 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

    4.2.1 Examples: Expectation . . . . . . . . . . . . . . . . 1924.3 Functions of Several Random Variables . . . . . . . . . . . . 2004.4 Properties of Expectation . . . . . . . . . . . . . . . . . . . 2004.5 Examples: Functions of Several Random Variables . . . . . 203

    4.5.1 Correlation . . . . . . . . . . . . . . . . . . . . . . . 2034.5.2 Covariance . . . . . . . . . . . . . . . . . . . . . . . 2054.5.3 Covariance Matrices . . . . . . . . . . . . . . . . . . 2064.5.4 Multivariable Characteristic Functions . . . . . . . . 2074.5.5 Example: Differential Entropy of a Gaussian Vector 209

    4.6 Conditional Expectation . . . . . . . . . . . . . . . . . . . . 2104.7 Jointly Gaussian Vectors . . . . . . . . . . . . . . . . . . . 213

  • CONTENTS ix

    4.8 Expectation as Estimation . . . . . . . . . . . . . . . . . . . 2164.9 Implications for Linear Estimation . . . . . . . . . . . . . 2224.10 Correlation and Linear Estimation . . . . . . . . . . . . . . 2244.11 Correlation and Covariance Functions . . . . . . . . . . . . 2314.12 The Central Limit Theorem . . . . . . . . . . . . . . . . . 2354.13 Sample Averages . . . . . . . . . . . . . . . . . . . . . . . . 2374.14 Convergence of Random Variables . . . . . . . . . . . . . . 2394.15 Weak Law of Large Numbers . . . . . . . . . . . . . . . . . 2444.16 Strong Law of Large Numbers . . . . . . . . . . . . . . . . 2464.17 Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . . 2514.18 Asymptotically Uncorrelated Processes . . . . . . . . . . . . 2564.19 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

    5 Second-Order Moments 2815.1 Linear Filtering of Random Processes . . . . . . . . . . . . 2825.2 Second-Order Linear Systems I/O Relations . . . . . . . . . 2845.3 Power Spectral Densities . . . . . . . . . . . . . . . . . . . . 2895.4 Linearly Filtered Uncorrelated Processes . . . . . . . . . . . 2925.5 Linear Modulation . . . . . . . . . . . . . . . . . . . . . . . 2985.6 White Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . 3015.7 Time-Averages . . . . . . . . . . . . . . . . . . . . . . . . . 3055.8 Differentiating Random Processes . . . . . . . . . . . . . . 3095.9 Linear Estimation and Filtering . . . . . . . . . . . . . . . 3125.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326

    6 A Menagerie of Processes 3436.1 Discrete Time Linear Models . . . . . . . . . . . . . . . . . 3446.2 Sums of IID Random Variables . . . . . . . . . . . . . . . . 3486.3 Independent Stationary Increments . . . . . . . . . . . . . . 3506.4 Second-Order Moments of ISI Processes . . . . . . . . . . 3536.5 Specification of Continuous Time ISI Processes . . . . . . . 3556.6 Moving-Average and Autoregressive Processes . . . . . . . . 3586.7 The Discrete Time Gauss-Markov Process . . . . . . . . . . 3606.8 Gaussian Random Processes . . . . . . . . . . . . . . . . . . 3616.9 The Poisson Counting Process . . . . . . . . . . . . . . . . 3616.10 Compound Processes . . . . . . . . . . . . . . . . . . . . . . 3646.11 Exponential Modulation . . . . . . . . . . . . . . . . . . . 3666.12 Thermal Noise . . . . . . . . . . . . . . . . . . . . . . . . . 3716.13 Ergodicity and Strong Laws of Large Numbers . . . . . . . 3736.14 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

  • x CONTENTS

    A Preliminaries 389A.1 Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 389A.2 Examples of Proofs . . . . . . . . . . . . . . . . . . . . . . . 397A.3 Mappings and Functions . . . . . . . . . . . . . . . . . . . . 401A.4 Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . 402A.5 Linear System Fundamentals . . . . . . . . . . . . . . . . . 405A.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410

    B Sums and Integrals 417B.1 Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . 417B.2 Double Sums . . . . . . . . . . . . . . . . . . . . . . . . . . 420B.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . 421B.4 The Lebesgue Integral . . . . . . . . . . . . . . . . . . . . 423

    C Common Univariate Distributions 427

    D Supplementary Reading 429

    Bibliography 434

    Index 438

  • Preface

    The origins of this book lie in our earlier book Random Processes: A Math-ematical Approach for Engineers, Prentice Hall, 1986. This book began asa second edition to the earlier book and the basic goal remains unchanged to introduce the fundamental ideas and mechanics of random processesto engineers in a way that accurately reflects the underlying mathematics,but does not require an extensive mathematical background and does notbelabor detailed general proofs when simple cases suffice to get the basicideas across. In the thirteen years since the original book was published,however, numerous improvements in the presentation of the material havebeen suggested by colleagues, students, teaching assistants, and by our ownteaching experience. The emphasis of the class shifted increasingly towardsexamples and a viewpoint that better reflected the course title: An Intro-duction to Statistical Signal Processing. Much of the basic content of thiscourse and of the fundamentals of random processes can be viewed as theanalysis of statistical signal processing systems: typically one is given aprobabilistic description for one random object, which can be consideredas an input signal. An operation or mapping or filtering is applied to theinput signal (signal processing) to produce a new random object, the out-put signal. Fundamental issues include the nature of the basic probabilisticdescription and the deri