Skip to content Skip to navigation

Connexions

You are here: Home » Content » The Cramer-Rao Lower Bound

Navigation

Content Actions
  • Download module PDF
  • Add to ...
    Add the module to:
    • My Favorites
    • A lens
    • An external social bookmarking service
    • My Favorites (What is 'My Favorites'?)
      'My Favorites' is a special kind of lens which you can use to bookmark modules and collections directly in Connexions. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need a Connexions account to use 'My Favorites'.
    • A lens (What is a lens?)

      Definition of a lens

      Lenses

      A lens is a custom view of Connexions content. You can think of it as a fancy kind of list that will let you see Connexions through the eyes of organizations and people you trust.

      What is in a lens?

      Lens makers point to Connexions materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

      Who can create a lens?

      Any individual Connexions member, a community, or a respected organization.

    • External bookmarks
  • E-mail the authors

Recently Viewed

The Cramer-Rao Lower Bound

Module by: Clayton Scott, Robert Nowak

The Cramer-Rao Lower Bound (CRLB) sets a lower bound on the variance of any unbiased estimator. This can be extremely useful in several ways:

  1. If we find an estimator that achieves the CRLB, then we know that we have found an MVUB estimator!
  2. The CRLB can provide a benchmark against which we can compare the performance of any unbiased estimator. (We know we're doing very well if our estimator is "close" to the CRLB.)
  3. The CRLB enables us to rule-out impossible estimators. That is, we know that it is physically impossible to find an unbiased estimator that beats the CRLB. This is useful in feasibility studies.
  4. The theory behind the CRLB can tell us if an estimator exists that achieves the bound.

Estimator Accuracy

Consider the likelihood function px|θ p θ x , where θθ is a scalar unknown (parameter). We can plot the likelihood as a function of the unknown, as shown in Figure 1.

Figure 1
Figure 1 ()
The more "peaky" or "spiky" the likelihood function, the easier it is to determind the unknown parameter.

Example 1

Suppose we observe x=A+w x A w where w∼σσ2 w σ σ 2 and AA is an unknown parameter. The "smaller" the noise w w is, the easier it will be to estimate AA from the observation xx.

Suppose A=3 A 3 and σ=1/3 σ 13 .

Figure 2
Figure 2 ()
Given this density function, we can easily rule-out estimates of AA greater than 4 or less than 2, since it is very unlikely that such AA could give rise to out observation.

On the other hand, suppose σ=1 σ 1 .

Figure 3
Figure 3 ()
In this case, it is very difficult to estimate AA. Since the noise power is larger, it is very difficult to distinguish AA from the noise.

The key thing to notice is that the estimation accuracy of AA depends on σσ, which in effect determines the peakiness of the likelihood. The more peaky, the better localized the data is about the true parameter.

To quantify the notion, note that the peakiness is effectively measured by the negative of the second derivative of the log-likelihood at its peak, as seen in Figure 4.

Figure 4
Figure 4 ()

Example 2

x=A+w x A w

logpx|A=-log2πσ2-12σ2x-A2 p A x 2 σ 2 1 2 σ 2 x A 2 (1)
∂∂Alogpx|A=1σ2x-A A p A x 1 σ 2 x A
-∂2∂A2logpx|A=1σ2 A 2 p A x 1 σ 2 (2)
The curvature increases as σ2 σ 2 decreases (curvature=peakiness).

In general, the curavture will depend on the observation data; -∂2∂θ2logpx|A θ 2 p A x is a function of xx. Therefore, an average measure of curvature is more appropriate.

-E∂2∂θ2logpx|θ θ 2 p θ x (3)
This average-out randomness due to the data and is a function of θθ alone.

We are now ready to state the CRLB theorem.

theorem 1: Cramer-Rao Lower Bound Theorem

Assume that the pdf px|θ p θ x satisfies the "regularity" condition ∀θ:E∂∂θlogpx|θ=0 θ θ p θ x 0 where the expectation is take with respect to px|θ p θ x . Then, the variance of any unbiased estimator θ ̂ θ must satisfy

σ θ ̂2≥1-E∂2∂θ2logpx|θ θ 1 θ 2 p θ x (4)
where the derivative is evaluated at the true value of θθ and the expectation is with respect to px|θ p θ x . Moreover, an unbiased estimator may be found that attains the bound for all θθ if and only if
∂∂θlogpx|θ=Iθgθ-θ θ p θ x I θ g θ θ (5)
for some functions gg and II.

The corresponding estimator is MVUB and is given by θ ̂=gx θ g x , and the minimum variance is 1Iθ 1 I θ .

Example

x=A+w x A w where w∼0σ2 w 0 σ 2 θ=A θ A ∀A:E∂∂θlogp=E1σ2x-A=0 A θ p 1 σ 2 x A 0 CRLB=1-E∂2∂θ2logp=11σ2=σ2 CRLB 1 θ 2 p 1 1 σ 2 σ 2 Therefore, any unbiased estimator A ̂ A has σ A ̂2≥σ2 A σ 2 . But we know that A ̂=x A x has σ A ̂2=σ2 A σ 2 . Therefore, A ̂=x A x is the MVUB estimator.

note:
θ=A θ A Iθ=1σ2 I θ 1 σ 2 gx=x g x x

Proof

First consider the reguarity condition: E∂∂θlogpx|θ=0 θ p θ x 0

note:
E∂∂θlogpx|θ=∫∂∂θlogpx|θpx|θdθ=∫∂∂θpx|θdθ θ p θ x θ θ p θ x p θ x θ θ p θ x
Now assuming that we can interchange order of differentiation and integration E∂∂θlogpx|θ=∂∂θ∫px|θdθ=∂∂θ1=0 θ p θ x θ θ p θ x θ 1 0 So the regularity condition is satisfied whenever this interchange is possible1; i.e., when derivative is well-defined, fails for uniform density.

Now lets derive the CRLB for a scalar parameter α=gθ α g θ , where the pdf is px|θ p θ x . Consider any unbiased estimator of α α: α ̂∈E α ̂=α=gθ α α α g θ Note that this is equivalent to ∫ α ̂px|θdx=gθ x α p θ x g θ where α ̂ α is unbiased. Now differentiate both side ∫ α ̂∂∂θpx|θdx=∂∂θgθ x α θ p θ x θ g θ or ∫ α ̂∂∂θlogpx|θpx|θdx=∂∂θgθ x α θ p θ x p θ x θ g θ

Now, exmploiting the regularity condition,

∫ α ̂-α∂∂θlogpx|θpx|θdx=∂∂θgθ x α α θ p θ x p θ x θ g θ (6)
since ∫α∂∂θlogpx|θpx|θdx=αElogpx|θ=0 x α θ p θ x p θ x α p θ x 0 Now apply the Cauchy-Schwarz inequality to the integral above: ∂∂θgθ2=∫ α ̂-α∂∂θlogpx|θpx|θdx2 θ g θ 2 x α α θ p θ x p θ x 2 ∂∂θgθ2≤∫ α ̂-α2px|θdx∫∂∂θlogpx|θpx|θdθ θ g θ 2 x α α 2 p θ x θ θ p θ x p θ x σ α ̂2 α is ∫ α ̂-α2px|θdx x α α 2 p θ x , so
σ α ̂2≥∂∂θgθ2E∂∂θlogpx|θ2 α θ g θ 2 θ p θ x 2 (7)
Now we note that E∂∂θlogpx|θ2=-E∂2∂θ2logpx|θ θ p θ x 2 θ 2 p θ x Why? Regularity condition. E∂∂θlogpx|θ=∫∂∂θlogpx|θpx|θdx=0 θ p θ x x θ p θ x p θ x 0 Thus, ∂∂θ∫∂∂θlogpx|θpx|θdx=0 θ x θ p θ x p θ x 0 or ∫∂2∂θ2logpx|θpx|θ+∂∂θlogpx|θ∂∂θpx|θdx=0 x θ 2 p θ x p θ x θ p θ x θ p θ x 0 Therefore, -E∂2∂θ2logpx|θ=∫∂∂θlogpx|θ∂∂θlogpx|θpx|θdx=E∂∂θlogpx|θ2 θ 2 p θ x x θ p θ x θ p θ x p θ x θ p θ x 2 Thus, Equation 7 becomes σ α ̂2≥∂∂θgθ2-E∂2∂θ2logpx|θ α θ g θ 2 θ 2 p θ x
note:
If gθ=θ g θ θ , then numerator is 1.

Example: DC Level in White Guassian Noise

∀n,n∈1…N: x n =A+ w n n n 1 … N x n A w n where w n ∼0σ2 w n 0 σ 2 px|A=12πσ2N2ⅇ-1σ2∑n=1N x n -A2 p A x 1 2 σ 2 N 2 1 σ 2 n 1 N x n A 2 ∂∂Alogpx|A=∂∂A-log2πσ2N2-12σ2∑n=1N x n -A2=1σ2∑n=1N x n -A A p A x A 2 σ 2 N 2 1 2 σ 2 n 1 N x n A 2 1 σ 2 n 1 N x n A E∂∂Alogpx|A=0 A p A x 0 ∂2∂A2logpx|A=-Nσ2 A 2 p A x N σ 2 Therefore, the variance of any unbiased estimator satisfies: σ A ̂2≥σ2N A