Cramer-Rao Bound 1.5 2003/06/04 2003/08/22 13:27:24.533 GMT-5 Don Johnson dhj@rice.edu Don Johnson dhj@rice.edu Kyle Clarkson kclarks@rice.edu Elizabeth Gregory lizzardg@rice.edu Mariyah Poonawala mariyah@rice.edu Matthew Jeanes mjeanes@rice.edu Jeffrey Silverman jsilv@rice.edu The mean-squared error for any estimate of a nonrandom parameter has a lower bound, the Cramér-Rao Bound (Cramér (1946) pp. 474-477), which defines the ultimate accuracy of any estimation procedure. This lower bound, as shown later, is intimately related to the maximum likelihood estimator. We seek a "bound" on the mean-squared error matrix M defined to be M θ θ θ θ ε ε A matrix is "lower bounded" by a second matrix if the difference between the two is a non-negative definite matrix. Define the column matrix x to be x θ θ b θ θ p r θ r where b θ denotes the column matrix of estimator biases. To derive the Cramér-Rao bound, evaluate x x . x x M b b I θ b I θ b F where θ b represents the matrix of partial derivatives of the bias θ j b i and the matrix F is the Fisher information matrix F θ p r θ r θ p r θ r Note that this matrix can alternatively be expressed as F ∇ θ ∇ θ T p r θ r The notation ∇ θ ∇ θ T means the matrix of all second partials of the quantity it operates on (the gradient of the gradient). The matrix is known as the Hessian. Demonstrating the equivalence of these two forms for the Fisher information is quite easy. Because r p r θ r 1 for all choices of the parameter vector, the gradient of the expression equals zero. Furthermore, θ p r θ r θ p r θ r p r θ r . Combining these results yields r θ p r θ r p r θ r 0 Evaluating the gradient for this quantity (using the chain rule) also yields zero. r ∇ θ ∇ θ T p r θ r p r θ r θ p r θ r θ p r θ r p r θ r 0 or θ p r θ r θ p r θ r ∇ θ ∇ θ T p r θ r Calculating the expected value for the Hessian for is somewhat easier than finding the expected value of the outer product of the gradient with itself. In the scalar case, we have θ p r θ r 2 θ 2 2 p r θ r Returning to the derivation, the matrix x x is non-negative definite because it is a correlation matrix. Thus, for any column matrix, α, the quadratic form α x x α is non-negative. Choose a form for α that simplifies the quadratic form. A convenient choice is α β F I θ b β where β is an arbitrary column matrix. The quadratic form becomes in this case α x x α β M b b I θ b F I θ b β As this quadratic form must be non-negative, the matrix expression enclosed in brackets must be non-negative definite. We thus obtain the well-known Cramér-Rao bound on the mean-square error matrix. ε ε b θ b θ I θ b F I θ b This form for the Cramér-Rao Bound does not mean that each term in the matrix of squared errors is greater than the corresponding term in the bounding matrix. As stated earlier, this expression means that the difference between these matrices is non-negative definite. For a matrix to be non-negative definite, each term on the main diagonal must be non-negative. The elements of the main diagonal of ε ε are the squared errors of the estimate of the individual parameters. Thus, for each parameter, the mean-squared estimation error can be no smaller than θ i θ i 2 b i θ 2 I θ b F I θ b i i This bound simplifies greatly if the estimator is unbiased ( b 0 ). In this case, the Cramér-Rao bound becomes θ i θ i 2 F i i Thus, the mean-squared error for each parameter in a multiple-parameter, unbiased-estimator problem can be no smaller than the corresponding diagonal term in the inverse for the Fisher information matrix. In such problems, the estimate's error characteristics of any parameter become intertwined with the other parameters in a complicated way. Any estimator satisfying the Cramér-Rao bound with equality is said to be efficient. Let's evaluate the Cramér-Rao bound for the example we have been discussing: the estimation of the mean and variance of a length L sequence of statistically independent Gaussian random variables. Let the estimate of the mean θ 1 be the sample average θ 1 r l L ; as shown in the last example, this estimate is unbiased. Let the estimate of the variance θ 2 be the unbiased estimate θ 2 r l θ 1 2 L 1 . Each term in the Fisher information matrix F is given by the expected value of the paired products of derivatives of the logarithm of the likelihood function. F i j θ i p r θ r θ j p r θ r The logarithm of the likelihood function is p r θ r L 2 2 θ 2 1 2 θ 2 l 0 L 1 r l θ 1 2 its partial derivatives are θ 1 p r θ r 1 θ 2 l 0 L 1 r l θ 1 θ 2 p r θ r L 2 θ 2 1 2 θ 2 2 l 0 L 1 r l θ 1 2 and its second partials are θ 1 2 2 p r θ r L θ 2 θ 1 θ 2 2 p r θ r 1 θ 2 2 l 0 L 1 r l θ 1 θ 2 θ 1 2 p r θ r 1 θ 2 2 l 0 L 1 r l θ 1 θ 2 2 2 p r θ r L 2 θ 2 2 1 θ 2 3 l 0 L 1 r l θ 1 2 The Fisher information matrix has the surprisingly simple form F L θ 2 0 0 L 2 θ 2 2 its inverse is also a diagonal matrix with the elements on the main diagonal equalling the reciprocal of those in the original matrix. Because of the zero-values off-diagonal entries in the Fisher information matrix, the errors between the corresponding estimates are not inter-dependent. In this problem, the mean-square estimation error can be no smaller than θ 1 θ 1 2 θ 2 L θ 2 θ 2 2 2 θ 2 2 L Note that nowhere in the preceding example did the form of the estimator enter into the computation of the bound. The only quantity used in the computation of the Cramér-Rao bound is the logarithm of the likelihood function, which is a consequence of the problem statement, not how it is solved. Only in the case of unbiased estimators is the bound independent of the estimators used.That's why we assumed in the example that we used an unbiased estimator for the variance. Because of this property, the Cramér-Rao bound is frequently used to assess the performance limits that can be obtained with an unbiased estimator in a particular problem. When bias is present, the exact form of the estimator's bias explicitly enters the computation of the bound. All too frequently, the unbiased form is used in situations where the existence of an unbiased estimator can be questioned. As we shall see, one such problem is time delay estimation, presumably of some importance to the reader. This misapplication of the unbiased Cramér-Rao arises from desperation: the estimator is so complicated and nonlinear that computing the bias is nearly impossible. As shown in this problem, biased estimators can yield mean-squared error smaller as well as larger than the unbiased version of the Cramér-Rao bound. Consequently, desperation can yield misinterpretation when a general result is misapplied. In the single-parameter estimation problem, the Cramér-Rao bound incorporating bias has the well-known form Note that this bound differs somewhat from that originally given by Cramér (1946) p.480; his derivation ignores the additive bias term b b . ε 2 b 2 1 θ b 2 θ p r θ r 2 Note that the sign of the bias's derivative determines whether this bound is larger or potentially smaller than the unbiased version, which is obtained by setting the bias term to zero.

Efficiency An interesting question arises: when, if ever, is the bound satisfied with equality? Recalling the details of the derivation of the bound, equality results when the quantity α x x α equals zero. As this quantity is the expected value of the square of α x , it can only equal zero if α x 0 . Substituting in the form of the column matrices α and x, equality in the Cramér-Rao bound results whenever θ p r θ r I θ b F θ r θ b This complicated expression means that only if estimation problems (as expressed by the a priori density have the form of the right side of this equation can the mean-squared error equal the Cramér-Rao bound. In particular, the gradient of the log likelihood function can only depend on the observations through the estimator. In all other problems, the Cramér-Rao bound is a lower bound but not a tight one no estimator can have error characteristics that equal it. In such cases, we have limited insight into ultimate limitations on estimation error size with the Cramér-Rao bound. However, consider the case where the estimator is unbiased ( b 0 ). In addition, note the maximum likelihood estimate occurs when the gradient of the logarithm of the likelihood function equals zero: θ p r θ r 0 when θ θ ML . In this case, the condition for equality in the Cramér-Rao bound becomes F θ θ ML 0 As the Fisher information matrix is positive-definite, we conclude that if the estimator equals the maximum likelihood estimator, equality in the Cramér-Rao bound can be satisfied with equality, only the maximum likelihood estimate will achieve it. To use estimation theoretic terminology, if an efficient estimate exists, it is the maximum likelihood estimate. This result stresses the importance of maximum likelihood estimates, despite the seemingly ad hoc manner by which they are defined. Consider the Gaussian example being examined so frequently in this section. The components of the gradient of the logarithm of the likelihood function were given earlier by and . These expressions can be rearranged to reveal θ 1 p r θ r θ 2 p r θ r L θ 2 1 L l l r l θ 1 L 2 θ 2 1 2 θ 2 2 l l r l θ 1 2 The first component, which corresponds to the estimate of the mean, is expressed in the form required for the existence of an efficient estimate. The second component--the partial with respect to the variance θ 2 --cannot be rewritten in a similar fashion. No unbiased, efficient estimate of the variance exits in this problem. The mean-squared error of the variance's unbiased estimate, but not the maximum likelihood estimate, is lower-bounded by 2 θ 2 2 L 1 2 . This error is strictly greater than the Cramér-Rao bound of 2 θ 2 2 L 2 . As no unbiased estimate of the variance can have a mean-squared error equal to the Cramér-Rao bound (no efficient estimate exists for the variance in the Gaussian problem), one presumes that the closeness of the error of our unbiased estimator to the bound implies that it possesses the smallest squared-error of any estimate. This presumption may, of course, be incorrect.

Properties of the Maximum Likelihood Estimator The maximum likelihood estimate is the most used estimation technique for nonrandom parameters. Not only because of its close linkage to the Cramér-Rao bound, but also because it has desirable asymptotic properties in the context of any problem (Cramér (1946) pp. 500-506). The maximum likelihood estimate is at least asymptotically unbiased. It may be unbiased for any number of observations (as in the estimation of the mean of a sequence of independent random variable) for some problems. The maximum likelihood estimate is consistent. The maximum likelihood estimates is asymptotically efficient. As more and more data are incorporated into an estimate, the Cramér-Rao bound accurately projects the best attainable error and the maximum likelihood estimate has those optimal characteristics. Asymptotically, the maximum likelihood estimate is distributed as a Gaussian random variable. Because of the previous properties, the mean asymptotically equals the parameter and the covariance matrix is L F θ . Most would agree that a "good" estimator should have these properties. What these results do not provide is assessment of how many observations are needed for the asymptotic results to apply to some specified degree of precision. Consequently, they should be used with caution; for instance, some other estimator may have a smaller mean-square error than the maximum likelihood for a modest number of observations.

H. Cramér Mathematical Methods of Statistics Princeton University Press 1946 Princeton, NJ