Wiley.com
Print this page Share

Statistical Signal Processing in Engineering

ISBN: 978-1-119-29397-2
608 pages
February 2018
Statistical Signal Processing in Engineering (1119293979) cover image

Description

A problem-solving approach to statistical signal processing for practicing engineers, technicians, and graduate students 

This book takes a pragmatic approach in solving a set of common problems engineers and technicians encounter when processing signals. In writing it, the author drew on his vast theoretical and practical experience in the field to provide a quick-solution manual for technicians and engineers, offering field-tested solutions to most problems engineers can encounter. At the same time, the book delineates the basic concepts and applied mathematics underlying each solution so that readers can go deeper into the theory to gain a better idea of the solution’s limitations and potential pitfalls, and thus tailor the best solution for the specific engineering application. 

Uniquely, Statistical Signal Processing in Engineering can also function as a textbook for engineering graduates and post-graduates. Dr. Spagnolini, who has had a quarter of a century of experience teaching graduate-level courses in digital and statistical signal processing methods, provides a detailed axiomatic presentation of the conceptual and mathematical foundations of statistical signal processing that will challenge students’ analytical skills and motivate them to develop new applications on their own, or better understand the motivation underlining the existing solutions.  

Throughout the book, some real-world examples demonstrate how powerful a tool statistical signal processing is in practice across a wide range of applications.

  • Takes an interdisciplinary approach, integrating basic concepts and tools for statistical signal processing
  • Informed by its author’s vast experience as both a practitioner and teacher
  • Offers a hands-on approach to solving problems in statistical signal processing
  • Covers a broad range of applications, including communication systems, machine learning, wavefield and array processing, remote sensing, image filtering and distributed computations
  • Features numerous real-world examples from a wide range of applications showing the mathematical concepts involved in practice
  • Includes MATLAB code of many of the experiments in the book

Statistical Signal Processing in Engineering is an indispensable working resource for electrical engineers, especially those working in the information and communication technology (ICT) industry. It is also an ideal text for engineering students at large, applied mathematics post-graduates and advanced undergraduates in electrical engineering, applied statistics, and pure mathematics, studying statistical signal processing.

See More

Table of Contents

List of Figures xvii

List of Tables xxiii

Preface xxv

List of Abbreviations xxix

How to Use the Book xxxi

About the Companion Website xxxiii

Prerequisites xxxv

Why are there so many matrixes in this book? xxxvii

1 Manipulations on Matrixes 1

1.1 Matrix Properties 1

1.1.1 Elementary Operations 2

1.2 Eigen-Decomposition 6

1.3 Eigenvectors in Everyday Life 9

1.3.1 Conversations in a Noisy Restaurant 9

1.3.2 Power Control in a Cellular System 12

1.3.3 Price Equilibrium in the Economy 14

1.4 Derivative Rules 15

1.4.1 Derivative with respect to x 16

1.4.2 Derivative with respect to x 17

1.4.3 Derivative with respect to the Matrix X 18

1.5 Quadratic Forms 19

1.6 Diagonalization of a Quadratic Form 20

1.7 Rayleigh Quotient 21

1.8 Basics of Optimization 22

1.8.1 Quadratic Function with Simple Linear Constraint (M=1) 23

1.8.2 Quadratic Function with Multiple Linear Constraints 23

Appendix A: Arithmetic vs. Geometric Mean 24

2 Linear Algebraic Systems 27

2.1 Problem Definition and Vector Spaces 27

2.1.1 Vector Spaces in Tomographic Radiometric Inversion 29

2.2 Rotations 31

2.3 Projection Matrixes and Data-Filtering 33

2.3.1 Projections and Commercial FM Radio 34

2.4 Singular Value Decomposition (SVD) and Subspaces 34

2.4.1 How to Choose the Rank of Afor Gaussian Model? 35

2.5 QR and Cholesky Factorization 36

2.6 Power Method for Leading Eigenvectors 38

2.7 Least Squares Solution of Overdetermined Linear Equations 39

2.8 Efficient Implementation of the LS Solution 41

2.9 Iterative Methods 42

3 Random Variables in Brief 45

3.1 Probability Density Function (pdf), Moments, and Other Useful Properties 45

3.2 Convexity and Jensen Inequality 49

3.3 Uncorrelatedness and Statistical Independence 49

3.4 Real-Valued Gaussian Random Variables 51

3.5 Conditional pdf for Real-Valued Gaussian Random Variables 54

3.6 Conditional pdf in Additive Noise Model 56

3.7 Complex Gaussian Random Variables 56

3.7.1 Single Complex Gaussian Random Variable 56

3.7.2 Circular Complex Gaussian Random Variable 57

3.7.3 Multivariate Complex Gaussian Random Variables 58

3.8 Sum of Square of Gaussians: Chi-Square 59

3.9 Order Statistics for N rvs 60

4 Random Processes and Linear Systems 63

4.1 Moment Characterizations and Stationarity 64

4.2 Random Processes and Linear Systems 66

4.3 Complex-Valued Random Processes 68

4.4 Pole-Zero and Rational Spectra (Discrete-Time) 69

4.4.1 Stability of LTI Systems 70

4.4.2 Rational PSD 71

4.4.3 Paley–Wiener Theorem 72

4.5 Gaussian Random Process (Discrete-Time) 73

4.6 Measuring Moments in Stochastic Processes 75

Appendix A: Transforms for Continuous-Time Signals 76

Appendix B: Transforms for Discrete-Time Signals 79

5 Models and Applications 83

5.1 Linear Regression Model 84

5.2 Linear Filtering Model 86

5.2.1 Block-Wise Circular Convolution 88

5.2.2 Discrete Fourier Transform and Circular Convolution Matrixes 89

5.2.3 Identification and Deconvolution 90

5.3 MIMO systems and Interference Models 91

5.3.1 DSL System 92

5.3.2 MIMO in Wireless Communication 92

5.4 Sinusoidal Signal 97

5.5 Irregular Sampling and Interpolation 97

5.5.1 Sampling With Jitter 100

5.6 Wavefield Sensing System 101

6 Estimation Theory 105

6.1 Historical Notes 105

6.2 Non-Bayesian vs. Bayesian 106

6.3 Performance Metrics and Bounds 107

6.3.1 Bias 107

6.3.2 Mean Square Error (MSE) 108

6.3.3 Performance Bounds 109

6.4 Statistics and Sufficient Statistics 110

6.5 MVU and BLU Estimators 111

6.6 BLUE for Linear Models 112

6.7 Example: BLUE of the Mean Value of Gaussian rvs 114

7 Parameter Estimation 117

7.1 Maximum Likelihood Estimation (MLE) 117

7.2 MLE for Gaussian Model 119

7.2.1 Additive Noise Model with 119

7.2.2 Additive Noise Model with 120

7.2.3 Additive Noise Model with Multiple Observations with Known 121

7.2.3.1 Linear Model 121

7.2.3.2 Model 122

7.2.3.3 Model 123

7.2.4 Model 123

7.2.5 Additive Noise Model with Multiple Observations with Unknown 124

7.3 Other Noise Models 125

7.4 MLE and Nuisance Parameters 126

7.5 MLE for Continuous-Time Signals 128

7.5.1 Example: Amplitude Estimation 129

7.5.2 MLE for Correlated Noise 130

7.6 MLE for Circular Complex Gaussian 131

7.7 Estimation in Phase/Frequency Modulations 131

7.7.1 MLE Phase Estimation 132

7.7.2 Phase Locked Loops 133

7.8 Least Square (LS) Estimation 135

7.8.1 Weighted LS with 136

7.8.2 LS Estimation and Linear Models 137

7.8.3 Under or Over-Parameterizing? 138

7.8.4 Constrained LS Estimation 139

7.9 Robust Estimation 140

8 Cramér–Rao Bound 143

8.1 Cramér–Rao Bound and Fisher Information Matrix 143

8.1.1 CRB for Scalar Problem (P=1) 143

8.1.2 CRB and Local Curvature of Log-Likelihood 144

8.1.3 CRB for Multiple Parameters (p 1) 144

8.2 Interpretation of CRB and Remarks 146

8.2.1 Variance of Each Parameter 146

8.2.2 Compactness of the Estimates 146

8.2.3 FIM for Known Parameters 147

8.2.4 Approximation of the Inverse of FIM 148

8.2.5 Estimation Decoupled From FIM 148

8.2.6 CRB and Nuisance Parameters 149

8.2.7 CRB for Non-Gaussian rv and Gaussian Bound 149

8.3 CRB and Variable Transformations 150

8.4 FIM for Gaussian Parametric Model 151

8.4.1 FIM for with 151

8.4.2 FIM for Continuous-Time Signals in Additive White Gaussian Noise 152

8.4.3 FIM for Circular Complex Model 152

Appendix A: Proof of CRB 154

Appendix B: FIM for Gaussian Model 156

Appendix C: Some Derivatives for MLE and CRB Computations 157

9 MLE and CRB for Some Selected Cases 159

9.1 Linear Regressions 159

9.2 Frequency Estimation 162

9.3 Estimation of Complex Sinusoid 164

9.3.1 Proper, Improper, and Non-Circular Signals 165

9.4 Time of Delay Estimation 166

9.5 Estimation of Max for Uniform pdf 170

9.6 Estimation of Occurrence Probability for Binary pdf 172

9.7 How to Optimize Histograms? 173

9.8 Logistic Regression 176

10 Numerical Analysis and Montecarlo Simulations 179

10.1 System Identification and Channel Estimation 181

10.1.1 Matlab Code and Results 184

10.2 Frequency Estimation 184

10.2.1 Variable (Coarse/Fine) Sampling 187

10.2.2 Local Parabolic Regression 189

10.2.3 Matlab Code and Results 190

10.3 Time of Delay Estimation 192

10.3.1 Granularity of Sampling in ToD Estimation 193

10.3.2 Matlab Code and Results 194

10.4 Doppler-Radar System by Frequency Estimation 196

10.4.1 EM Method 197

10.4.2 Matlab Code and Results 199

11 Bayesian Estimation 201

11.1 Additive Linear Model with Gaussian Noise 203

11.1.1 Gaussian A-priori: 204

11.1.2 Non-Gaussian A-Priori 206

11.1.3 Binary Signals: MMSE vs. MAP Estimators 207

11.1.4 Example: Impulse Noise Mitigation 210

11.2 Bayesian Estimation in Gaussian Settings 212

11.2.1 MMSE Estimator 213

11.2.2 MMSE Estimator for Linear Models 213

11.3 LMMSE Estimation and Orthogonality 215

11.4 Bayesian CRB 218

11.5 Mixing Bayesian and Non-Bayesian 220

11.5.1 Linear Model with Mixed Random/Deterministic Parameters 220

11.5.2 Hybrid CRB 222

11.6 Expectation-Maximization (EM) 223

11.6.1 EM of the Sum of Signals in Gaussian Noise 224

11.6.2 EM Method for the Time of Delay Estimation of Multiple Waveforms 227

11.6.3 Remarks 228

Appendix A: Gaussian Mixture pdf 229

12 Optimal Filtering 231

12.1 Wiener Filter 231

12.2 MMSE Deconvolution (or Equalization) 233

12.3 Linear Prediction 234

12.3.1 Yule–Walker Equations 235

12.4 LS Linear Prediction 237

12.5 Linear Prediction and AR Processes 239

12.6 Levinson Recursion and Lattice Predictors 241

13 Bayesian Tracking and Kalman Filter 245

13.1 Bayesian Tracking of State in Dynamic Systems 246

13.1.1 Evolution of the A-posteriori pdf 247

13.2 Kalman Filter (KF) 249

13.2.1 KF Equations 251

13.2.2 Remarks 253

13.3 Identification of Time-Varying Filters in Wireless Communication 255

13.4 Extended Kalman Filter (EKF) for Non-Linear Dynamic Systems 257

13.5 Position Tracking by Multi-Lateration 258

13.5.1 Positioning and Noise 260

13.5.2 Example of Position Tracking 263

13.6 Non-Gaussian Pdf and Particle Filters264

14 Spectral Analysis 267

14.1 Periodogram 268

14.1.1 Bias of the Periodogram 268

14.1.2 Variance of the Periodogram 271

14.1.3 Filterbank Interpretation 273

14.1.4 Pdf of the Periodogram (White Gaussian Process) 274

14.1.5 Bias and Resolution 275

14.1.6 Variance Reduction and WOSA 278

14.1.7 Numerical Example: Bandlimited Process and (Small) Sinusoid 280

14.2 Parametric Spectral Analysis 282

14.2.1 MLE and CRB 284

14.2.2 General Model for AR, MA, ARMA Spectral Analysis 285

14.3 AR Spectral Analysis 286

14.3.1 MLE and CRB 286

14.3.2 A Good Reason to Avoid Over-Parametrization in AR 289

14.3.3 Cramér–Rao Bound of Poles in AR Spectral Analysis 291

14.3.4 Example: Frequency Estimation by AR Spectral Analysis 293

14.4 MA Spectral Analysis 296

14.5 ARMA Spectral Analysis 298

14.5.1 Cramér–Rao Bound for ARMA Spectral Analysis 300

Appendix A: Which Sample Estimate of the Autocorrelation to Use? 302

Appendix B: Eigenvectors and Eigenvalues of Correlation Matrix 303

Appendix C: Property of Monic Polynomial 306

Appendix D: Variance of Pole in AR(1) 307

15 Adaptive Filtering 309

15.1 Adaptive Interference Cancellation 311

15.2 Adaptive Equalization in Communication Systems 313

15.2.1 Wireless Communication Systems in Brief 313

15.2.2 Adaptive Equalization 315

15.3 Steepest Descent MSE Minimization 317

15.3.1 Convergence Analysis and Step-Size 318

15.3.2 An Intuitive View of Convergence Conditions 320

15.4 From Iterative to Adaptive Filters 323

15.5 LMS Algorithm and Stochastic Gradient 324

15.6 Convergence Analysis of LMS Algorithm 325

15.6.1 Convergence in the Mean 326

15.6.2 Convergence in the Mean Square 326

15.6.3 Excess MSE 329

15.7 Learning Curve of LMS 331

15.7.1 Optimization of the Step-Size 332

15.8 NLMS Updating and Non-Stationarity 333

15.9 Numerical Example: Adaptive Identification 334

15.10 RLS Algorithm 338

15.10.1 Convergence Analysis 339

15.10.2 Learning Curve of RLS 341

15.11 Exponentially-Weighted RLS 342

15.12 LMS vs. RLS 344

Appendix A: Convergence in Mean Square 344

16 Line Spectrum Analysis 347

16.1 Model Definition 349

16.1.1 Deterministic Signals 350

16.1.2 Random Signals 350

16.1.3 Properties of Structured Covariance 351

16.2 Maximum Likelihood and Cramér–Rao Bounds 352

16.2.1 Conditional ML 353

16.2.2 Cramér–Rao Bound for Conditional Model 354

16.2.3 Unconditional ML 356

16.2.4 Cramér–Rao Bound for Unconditional Model 356

16.2.5 Conditional vs. Unconditional Model & Bounds 357

16.3 High-Resolution Methods 357

16.3.1 Iterative Quadratic ML (IQML) 358

16.3.2 Prony Method 360

16.3.3 MUSIC 360

16.3.4 ESPRIT 363

16.3.5 Model Order 365

17 Equalization in Communication Engineering 367

17.1 Linear Equalization 369

17.1.1 Zero Forcing (ZF) Equalizer 370

17.1.2 Minimum Mean Square Error (MMSE) Equalizer 371

17.1.3 Finite-Length/Finite-Block Equalizer 371

17.2 Non-Linear Equalization 372

17.2.1 ZF-DFE 373

17.2.2 MMSE–DFE 374

17.2.3 Finite-Length MMSE–DFE 375

17.2.4 Asymptotic Performance for Infinite-Length Equalizers 376

17.3 MIMO Linear Equalization 377

17.3.1 ZF MIMO Equalization 377

17.3.2 MMSE MIMO Equalization 379

17.4 MIMO–DFE Equalization 379

17.4.1 Cholesky Factorization and Min/Max Phase Decomposition 379

17.4.2 MIMO–DFE 380

18 2D Signals and Physical Filters 383

18.1 2D Sinusoids 384

18.1.1 Moiré Pattern 386

18.2 2D Filtering 388

18.2.1 2D Random Fields 390

18.2.2 Wiener Filtering 391

18.2.3 Image Acquisition and Restoration 392

18.3 Diffusion Filtering 394

18.3.1 Evolution vs. Time: Fourier Method 394

18.3.2 Extrapolation of the Density 395

18.3.3 Effect of Phase-Shift 396

18.4 Laplace Equation and Exponential Filtering 398

18.5 Wavefield Propagation 400

18.5.1 Propagation/Backpropagation 400

18.5.2 Wavefield Extrapolation and Focusing 402

18.5.3 Exploding Reflector Model 402

18.5.4 Wavefield Extrapolation 404

18.5.5 Wavefield Focusing (or Migration) 406

Appendix A: Properties of 2D Signals 406

Appendix B: Properties of 2D Fourier Transform 410

Appendix C: Finite Difference Method for PDE-Diffusion 412

19 Array Processing 415

19.1 Narrowband Model 415

19.1.1 Multiple DoAs and Multiple Sources 419

19.1.2 Sensor Spacing Design 420

19.1.3 Spatial Resolution and Array Aperture 421

19.2 Beamforming and Signal Estimation 422

19.2.1 Conventional Beamforming 425

19.2.2 Capon Beamforming (MVDR) 426

19.2.3 Multiple-Constraint Beamforming 429

19.2.4 Max-SNR Beamforming 431

19.3 DoA Estimation 432

19.3.1 ML Estimation and CRB 433

19.3.2 Beamforming and Root-MVDR 434

20 Multichannel Time of Delay Estimation 435

20.1 Model Definition for ToD 440

20.2 High Resolution Method for ToD (L=1) 441

20.2.1 ToD in the Fourier Transformed Domain 441

20.2.2 CRB and Resolution 444

20.3 Difference of ToD (DToD) Estimation 445

20.3.1 Correlation Method for DToD 445

20.3.2 Generalized Correlation Method 448

20.4 Numerical Performance Analysis of DToD 452

20.5 Wavefront Estimation: Non-Parametric Method (L=1) 454

20.5.1 Wavefront Estimation in Remote Sensing and Geophysics 456

20.5.2 Narrowband Waveforms and 2D Phase Unwrapping 457

20.5.3 2D Phase Unwrapping in Regular Grid Spacing 458

20.6 Parametric ToD Estimation and Wideband Beamforming 460

20.6.1 Delay and Sum Beamforming 462

20.6.2 Wideband Beamforming After Fourier Transform 464

20.7 Appendix A: Properties of the Sample Correlations 465

20.8 Appendix B: How to Delay a Discrete-Time Signal? 466

20.9 Appendix C: Wavefront Estimation for 2D Arrays 467

21 Tomography 467

21.1 X-ray Tomography 471

21.1.1 Discrete Model 471

21.1.2 Maximum Likelihood 473

21.1.3 Emission Tomography 473

21.2 Algebraic Reconstruction Tomography (ART) 475

21.3 Reconstruction From Projections: Fourier Method 475

21.3.1 Backprojection Algorithm 476

21.3.2 How Many Projections to Use? 479

21.4 Traveltime Tomography 480

21.5 Internet (Network) Tomography 483

21.5.1 Latency Tomography 484

21.5.2 Packet-Loss Tomography 484

22 Cooperative Estimation 487

22.1 Consensus and Cooperation 490

22.1.1 Vox Populi: The Wisdom of Crowds 490

22.1.2 Cooperative Estimation as Simple Information Consensus 490

22.1.3 Weighted Cooperative Estimation ( ) 493

22.1.4 Distributed MLE ( ) 495

22.2 Distributed Estimation for Arbitrary Linear Models (p>1) 496

22.2.1 Centralized MLE 497

22.2.2 Distributed Weighted LS 498

22.2.3 Distributed MLE 500

22.2.4 Distributed Estimation for Under-Determined Systems 501

22.2.5 Stochastic Regressor Model 503

22.2.6 Cooperative Estimation in the Internet of Things (IoT) 503

22.2.7 Example: Iterative Distributed Estimation 505

22.3 Distributed Synchronization 506

22.3.1 Synchrony-States for Analog and Discrete-Time Clocks 507

22.3.2 Coupled Clocks 510

22.3.3 Internet Synchronization and the Network Time Protocol (NTP) 512

Appendix A: Basics of Undirected Graphs 515

23 Classification and Clustering 521

23.1 Historical Notes 522

23.2 Classification 523

23.2.1 Binary Detection Theory 523

23.2.2 Binary Classification of Gaussian Distributions 528

23.3 Classification of Signals in Additive Gaussian Noise 529

23.3.1 Detection of Known Signal 531

23.3.2 Classification of Multiple Signals 532

23.3.3 Generalized Likelihood Ratio Test (GLRT) 533

23.3.4 Detection of Random Signals 535

23.4 Bayesian Classification 536

23.4.1 To Classify or Not to Classify? 537

23.4.2 Bayes Risk 537

23.5 Pattern Recognition and Machine Learning 538

23.5.1 Linear Discriminant 539

23.5.2 Least Squares Classification 540

23.5.3 Support Vectors Principle 541

23.6 Clustering 543

23.6.1 K-Means Clustering 544

23.6.2 EM Clustering 545

References 549

Index 557

See More

Author Information

UMBERTO SPAGNOLINI is Professor in Signal Processing and Telecommunications at Politecnico di Milano, Italy. Prof. Spagnolini's research focuses on statistical signal processing, communication systems, and advanced topics in signal processing for remote sensing and wireless communication systems. He is a Senior Member of the IEEE, engages in editorial activity for IEEE journals and conferences, and has authored 300 patents and papers in peer reviewed journals and conferences.

See More

Related Titles

Back to Top