Introduction to random variables and processes. Concept of estimator. Minimum variance unbiased estimator. Cramer-Rao lower bound. Sufficient statistics. Linear models and estimators. Maximum likelihood estimation. Least Square estimation. Bayesian approach to estimation, MMSE and MAP estimators, Wiener filtering. Nonparametric spectral estimation. Parametric spectral estimation. Rational spectra, ARMA models. Line spectra, subspace methods. Spatial estimation, sensor arrays.
1) S.M. Kay, Fundamentals of statistical signal processing: Volume I - Estimation theory, Prentice Hall, 1998.
2) A.M. Mood, F.A. Graybill, D.C. Boes, Introduction to the theory of statistics, McGraw-Hill, 1974.
3) L.L Sharf, Statistical Signal Processing, Addison-Wesley Publishing Co., 1990.
4) M.H. Hayes, Statistical Digital Signal Processing and Modeling, John Wiley & Sons, 1996.
5) P. Stoica, R.L. Moses, Spectral Analysis of Signals, Prentice Hall, 2005.
6) A. Papoulis, S.U. Pillai, Probability, Random Variables, and Stochastic Processes, 4th ed., McGraw-Hill, 2002.
7) D. Manolakis, V.K. Ingle, S.M. Kogon, Statistical and Adaptive Signal Processing: Spectral Estimation, Signal Modeling, Adaptive Filtering and Array Processing, Artech House, 2005.
Learning Objectives
The course aims to provide the basic knowledge for the treatment of stochastic signals (with particular attention to the theory of parameter estimation, the filtering of random signals, the methods of spectral estimation) and the ability to apply this knowledge in concrete cases in the field of Telecommunications Engineering.
Knowledge and understanding:
- knowledge of the classical methods of estimation theory (minimum variance unbiased estimators, sufficient statistics, Cramer-Rao lower bound, maximum likelihood estimation, least squares estimation);
- knowledge of Bayesian estimation methods (MMSE, MAP, LMMSE, Wiener filtering);
- knowledge of non-parametric spectral estimation methods (periodogram, Welch-Bartlett, Blackman-Tukey);
- knowledge of parametric spectral estimation methods (ARMA models, subspace methods, MUSIC, ESPRIT).
Ability to apply knowledge and understanding:
- ability to classify the different methods and criteria used in the theory of estimation and to select the most suitable methods in the single applications to extract the parameters of interest of a signal in the presence of noise;
- ability to implement estimation algorithms in application contexts in the field of Telecommunications Engineering through suitable simulation software and to evaluate their performance.
The course aims at providing basic knowledge for the treatment of stochastic signals, with particular emphasis to the theory of parameter estimation (by using both classical and Bayesian approaches), the filtering of random signals, the methods for spectral estimation.
At the end of the course, the student will acquire:
- knowledge of classical methods of estimation theory (unbiased minimum variance estimators, sufficient statistic, Cramer-Rao lower bound, maximum likelihood estimators, least squares), and those of Bayesian estimation (MMSE, MAP, LMMSE, Wiener filtering);
- knowledge of parametric spectral estimation methods (ARMA models, subspaces methods, MUSIC, ESPRIT) and of non-parametric ones (periodogram, Welch-Bartlett, Blackman-Tukey);
- ability to classify the various methods and criteria used in estimation theory and to select the algorithms more suitable to extract the parameters of interest from a signal, in a given application, in the presence of noise.
Prerequisites
The student is expected to have a basic knowledge about: signals and systems; probability; random variables and processes and their characterization in the time and frequency domains; vector and matrix representations.
Teaching Methods
Lectures
Type of Assessment
The final exam consists of two parts:
- a computer project to be solved (preferably) in MATLAB on a topic agreed with the teacher;
- an oral exam on the topics developed during the course.
The objective of the first test (computer project) is to verify:
- the ability to model data in an estimation problem;
- the ability to extract information of interest by using one of the estimators discussed during the course;
- the ability to use appropriate quality criteria to evaluate the performance of an estimator.
The aim of the oral test is to verify:
- the theoretical knowledge at the basis of classical and Bayesian estimation;
- the theoretical knowledge at the basis of spectral estimation.
Course program
Review of random variables and random processes. Density and cumulative probability functions. Expectations, moments, joint moments. Gaussian PDF. Stationary processes. Autocorrelation and autocovariance matrices. Power spectral density. Filtering of random processes. Spectral factorization theorem. Introduction to the estimation problem. Observed data and signal models. PDF of data. Bias and unbiased estimators. Minimum variance unbiased (MVU) estimators. Cramer-Rao lower bound (CRLB). Fisher information. CRLB and transformation of parameters. CRLB for signals in AWGN. Sufficient statistics. Neyman-Fischer factorization. Rao-Blackwell-Lehmann-Scheffe Theorem. Linear signal model. Estimator for linear signal model and its covariance. Generalized linear signal model. Best linear unbiased estimator (BLUE). Maximum likelihood estimator (MLE). Asymptotic properties of the MLE. Numeric computation of the MLE. MLE for a vector of parameters. Least squares estimator (LS). LS estimator with a linear model. Weighted LS. Geometrical interpretation of the LS estimator. Bayesian approach to estimation. Prior and posterior PDF. MMSE Bayesian estimator. generalized linear Bayesian model. Examples of Bayesian estimation. MMSE Bayesian estimation for a vector of parameters. Bayesian risk. Maximum a posteriori (MAP) estimation, scalar and vector of parameters cases. Examples of MAP estimation. Linear MMSE (LMMSE) estimation. Geometric interpretation of the LMMSE estimation. Wiener filtering, prediction and smoothing. Spectral estimation. Estimation of the autocorrelation sequenze. Periodogram. Performances of periodogram estimates: average and variance. Modified periodogram. Methods of Welch and Bartlett. Method of Blackman-Tukey. Performances of periodogram-averaging and periodogram-smoothing methods. Parametric methods: spectral estimation based on AR, MA and ARMA models. Subspaces methods: Pisarenko Harmonic Decomposition, MUSIC, ESPRIT. Spatial methods and sensor arrays.