Estimator Accuracy Consider the likelihood function p θ x , where θ is a scalar unknown (parameter). We can plot the likelihood as a function of the unknown, as shown in .

The more "peaky" or "spiky" the likelihood function, the easier it is to determind the unknown parameter. Suppose we observe x A w where w σ σ 2 and A is an unknown parameter. The "smaller" the noise w is, the easier it will be to estimate A from the observation x. Suppose A 3 and σ 13 .

Given this density function, we can easily rule-out estimates of A greater than 4 or less than 2, since it is very unlikely that such A could give rise to out observation. On the other hand, suppose σ 1 .

In this case, it is very difficult to estimate A. Since the noise power is larger, it is very difficult to distinguish A from the noise. The key thing to notice is that the estimation accuracy of A depends on σ, which in effect determines the peakiness of the likelihood. The more peaky, the better localized the data is about the true parameter. To quantify the notion, note that the peakiness is effectively measured by the negative of the second derivative of the log-likelihood at its peak, as seen in .

x A w p A x 2 σ 2 1 2 σ 2 x A 2 A p A x 1 σ 2 x A A 2 p A x 1 σ 2 The curvature increases as σ 2 decreases (curvature=peakiness). In general, the curavture will depend on the observation data; θ 2 p A x is a function of x. Therefore, an average measure of curvature is more appropriate. θ 2 p θ x This average-out randomness due to the data and is a function of θ alone. We are now ready to state the CRLB theorem. Cramer-Rao Lower Bound Theorem Assume that the pdf p θ x satisfies the "regularity" condition θ θ p θ x 0 where the expectation is take with respect to p θ x . Then, the variance of any unbiased estimator θ must satisfy θ 1 θ 2 p θ x where the derivative is evaluated at the true value of θ and the expectation is with respect to p θ x . Moreover, an unbiased estimator may be found that attains the bound for all θ if and only if θ p θ x I θ g θ θ for some functions g and I. The corresponding estimator is MVUB and is given by θ g x , and the minimum variance is 1 I θ . x A w where w 0 σ 2 θ A A θ p 1 σ 2 x A 0 CRLB 1 θ 2 p 1 1 σ 2 σ 2 Therefore, any unbiased estimator A has A σ 2 . But we know that A x has A σ 2 . Therefore, A x is the MVUB estimator. θ A I θ 1 σ 2 g x x First consider the reguarity condition: θ p θ x 0 θ p θ x θ θ p θ x p θ x θ θ p θ x Now assuming that we can interchange order of differentiation and integration θ p θ x θ θ p θ x θ 1 0 So the regularity condition is satisfied whenever this interchange is possibleThis is simply the Fundamental Theorem of Calculus applied to p θ x . So long as p θ x is absolutely continuous with respect to the Lebesgue measure, this is possible.; i.e., when derivative is well-defined, fails for uniform density. Now lets derive the CRLB for a scalar parameter α g θ , where the pdf is p θ x . Consider any unbiased estimator of α : α α α g θ Note that this is equivalent to x α p θ x g θ where α is unbiased. Now differentiate both side x α θ p θ x θ g θ or x α θ p θ x p θ x θ g θ Now, exmploiting the regularity condition, x α α θ p θ x p θ x θ g θ since x α θ p θ x p θ x α p θ x 0 Now apply the Cauchy-Schwarz inequality to the integral above: θ g θ 2 x α α θ p θ x p θ x 2 θ g θ 2 x α α 2 p θ x θ θ p θ x p θ x α is x α α 2 p θ x , so α θ g θ 2 θ p θ x 2 Now we note that θ p θ x 2 θ 2 p θ x Why? Regularity condition. θ p θ x x θ p θ x p θ x 0 Thus, θ x θ p θ x p θ x 0 or x θ 2 p θ x p θ x θ p θ x θ p θ x 0 Therefore, θ 2 p θ x x θ p θ x θ p θ x p θ x θ p θ x 2 Thus, becomes α θ g θ 2 θ 2 p θ x If g θ θ , then numerator is 1. DC Level in White Guassian Noise n n 1 … N x n A w n where w n 0 σ 2 p A x 1 2 σ 2 N 2 1 σ 2 n 1 N x n A 2 A p A x A 2 σ 2 N 2 1 2 σ 2 n 1 N x n A 2 1 σ 2 n 1 N x n A A p A x 0 A 2 p A x N σ 2 Therefore, the variance of any unbiased estimator satisfies: A σ 2 N The sample-mean estimator A 1 N n 1 N x n attains this bound and therefore is MVUB. When the CRLB is attained θ 1 I θ where I θ θ 2 p θ x The quantity I θ is called Fisher Information that x contains about θ. By CRLB Theorem, θ 1 θ 2 p θ x and θ p θ x I θ θ θ This yields θ 2 p θ x θ I θ θ θ I θ which in turn yields θ 2 p θ x I θ So, θ 1 I θ The CRLB is not always attained. Phase Estimation n n 1 … N x n A 2 f 0 n φ w n The amplitude and frequency are assumed known w n 0 σ 2 idd. p φ x 1 2 σ 2 N 2 1 2 σ 2 n 1 N x n A 2 f 0 n φ φ p φ x A σ 2 n 1 N x n 2 f 0 n φ A 2 4 f 0 n φ φ 2 p φ x A σ 2 n 1 N x n 2 f 0 n φ A 2 f 0 n 2 φ φ 2 p φ x A 2 σ 2 n 1 N 12 12 4 f 0 n 2 φ 4 f 0 n 2 φ Since I φ φ 2 p φ x , I φ N A 2 2 σ 2 because f 0 0 f 0 k 1 N 4 f 0 n 0 Therefore, φ 2 σ 2 N A 2 In this case, it can be shown that there does not exist a g such that φ p φ x I φ g x φ Therefore, an unbiased phase estimator that attains the CRLB does not exist. However, a MVUB estimator may still exist--only its variance will be larger than the CRLB.

CRLB θ i I θ i i where i j I θ i j θ i θ j p θ x I θ is the Fisher Information Matrix. Cramer-Rao Lower Bound - Vector Parameter Assume the pdf p φ x satisfies the "regularity" condition θ θ p θ x 0 Then the convariance matrix of any unbiased estimator θ satisfies C θ ^ I θ 0 (meaning C θ ^ I θ is p.s.d.) The Fisher Information matrix is I θ i j θ 2 p θ x Furthermore, θ attains the CRLB ( C θ ^ I θ ) iff θ p θ x I θ g x θ and θ g x DC Level in White Guassian Noise n n 1 … N x n A w n A is unknown and w n 0 σ 2 , where σ 2 is unknown. θ A σ 2 p θ x N 2 2 N 2 σ 2 1 2 σ 2 n 2 N x n A 2 A p θ x 1 σ 2 n 1 N x n A σ 2 p θ x N σ 2 1 2 σ 4 n 1 N x n A 2 A 2 p θ x N σ 2 N σ 2 A σ 2 p θ x 1 σ 4 n 1 N x n A 0 σ 2 2 p θ x N 2 σ 4 1 σ 6 n 1 N x n A 2 N 2 σ 4 Which leads to I θ N σ 2 0 0 N 2 σ 4 A σ 2 N σ 2 2 σ 4 N Note that the CRLB for A is the same whether or not σ 2 is known. This happens in this case due to the diagonal nature of the Fisher Information Matrix. In general the Fisher Information Matrix is not diagonal and consequently the CRLBs will depend on other unknown parameters.