Minimum Mean Squared Error Estimators 1.4 2003/05/15 2004/11/20 10:15:21.833 US/Central Don Johnson dhj@rice.edu Jeffrey Silverman jsilv@rice.edu Kevin Duh kevinduh@rice.edu Elizabeth Gregory lizzardg@rice.edu Don Johnson dhj@rice.edu Eileen Krause erkrause@rice.edu Mariyah Poonawala mariyah@rice.edu Matthew Jeanes mjeanes@rice.edu Kyle Clarkson kclarks@rice.edu Minimum Mean Squared Error MMSE estimate In terms of the densities involved in scalar random-parameter problems, the mean-squared error is given by ε 2 θ r θ θ 2 p r θ where p r θ is the joint density of the observations and the parameter. To minimize this integral with respect to θ , we rewrite using the laws of conditional probability as ε 2 r p r θ θ θ r 2 p r θ The density p r ( · ) is nonnegative. To minimize the mean-squared error, we must minimize the inner integral for each value of r because the integral is weighted by a positive quantity. We focus attention on the inner integral, which is the conditional expected value of the squared estimation error. The condition, a fixed value of r, implies that we seek that constant θ r derived from r that minimizes the second moment of the random parameter θ. A well-known result from probability theory states that the minimum of x c 2 occurs when the constant c equals the expected value of the random variable x (see Expected Values of Probability Functions). The inner integral and thereby the mean-squared error is minimized by choosing the estimator to be the conditional expected value of the parameter given the observations. θ MMSE r r θ Thus, a parameter's minimum mean-squared error (MMSE) estimate is the parameter's a posteriori (after the observations have been obtained) expected value. The associated conditional probability density p r θ is not often directly stated in a problem definition and must somehow be derived. In many applications, the likelihood function p θ r and the a priori density of the parameter are a direct consequence of the problem statement. These densities can be used to find the joint density of the observations and the parameter, enabling us to use Bayes's Rule to fine the a posteriori density if we knew the unconditional probability density of the observations. p r θ p θ r p θ p r This density p r is often difficult to determine. Be that as it may, to find the a posteriori conditional expected value, it need not be known. The numerator entirely expresses the a posteriori density's dependence on θ; the denominator only serves as the scaling factor to yield a unit-area quantity. The expected value is the center-of-mass of the probability density and does not depend directly on the "weight" of the density, bypassing calculation of the scaling factor. If not, the MMSE estimate can be exceedingly difficult to compute. Let L statistically independent observations be obtained, each of which is expressed by r l θ n l . Each n l is a Gaussian random variable having zero mean and variance σ n 2 . Thus, the unknown parameter in this problem is the mean of the observations. Assume it to be a Gaussian random variable a priori (mean m θ and variance σ θ 2 ). The likelihood function is easily found to be p θ r l 0 L 1 1 2 σ n 2 1 2 r l θ σ n 2 so that the a posteriori density is given by p r θ 1 2 σ θ 2 1 2 θ m θ σ θ 2 l 0 L 1 1 2 σ n 2 1 2 r l θ σ n 2 p r In an attempt to find the expected value of this distribution, lump all terms that do not depend explicitly on the quantity θ into a proportionality term. ∝ p r θ 1 2 r l θ 2 σ n 2 θ m θ 2 σ θ 2 After some manipulation, this expression can be written as ∝ p r θ 1 2 σ 2 θ σ 2 m θ σ θ 2 r l σ n 2 2 where σ 2 is a quantity that succinctly expresses the ratio σ n 2 σ θ 2 σ n 2 L σ θ 2 . The form of the a posteriori density suggests that it too is Gaussian; its mean, and therefore the MMSE estimate of θ, is given by θ MMSE r σ 2 m θ σ θ 2 r l σ n 2 More insight into the nature of this estimate is gained by rewriting it as θ MMSE r σ n 2 L σ θ 2 σ n 2 L m θ σ θ 2 σ θ 2 σ n 2 L 1 L l 0 L 1 r l The term σ n 2 L is the variance of the averaged observations for a given value of θ; it expresses the squared error encountered in estimating the mean by simple averaging. If this error is much greater than the a priori variance of θ ( ≫ σ n 2 L σ θ 2 ), implying that the observations are noisier than the variation of the parameter, the MMSE estimate ignores the observations and tends to yield the a priori mean m θ as its value. If the averaged observations are less variable than the parameter, the second term dominates, and the average of the observations is the estimate's value. This estimate behavior between these extremes is very intuitive. The detailed form of the estimate indicates how the squared error can be minimized by a linear combination of these extreme estimates. The conditional expected value of the estimate equals θ θ MMSE σ n 2 L σ θ 2 σ n 2 L m θ σ θ 2 σ θ 2 σ n 2 L θ This estimate is biased because its expected value does not equal the value of the sought-after parameter. It is asymptotically unbiased as the squared measurement error σ n 2 L tends to zero as L becomes large. The consistency of the estimator is determined by investigating the expected value of the squared error. Note that the variance of the a posteriori density is the quantity σ 2 ; as this quantity does not depend on r, it also equals the unconditional variance. As the number of observations increases, this variance tends to zero. In concert with the estimate being asymptotically unbiased, the expected value of the estimation error thus tends to zero, implying that we have a consistent estimate.