Skip to content Skip to navigation

Connexions

You are here: Home » Content » The Cramer-Rao Lower Bound

Navigation

Content Actions
  • Download module PDF
  • Add to ...
    Add the module to:
    • My Favorites
    • A lens
    • An external social bookmarking service
    • My Favorites (What is 'My Favorites'?)
      'My Favorites' is a special kind of lens which you can use to bookmark modules and collections directly in Connexions. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need a Connexions account to use 'My Favorites'.
    • A lens (What is a lens?)

      Definition of a lens

      Lenses

      A lens is a custom view of Connexions content. You can think of it as a fancy kind of list that will let you see Connexions through the eyes of organizations and people you trust.

      What is in a lens?

      Lens makers point to Connexions materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

      Who can create a lens?

      Any individual Connexions member, a community, or a respected organization.

      What are tags? tag icon

      Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

    • External bookmarks
  • E-mail the authors
  • Rate this module (How does the rating system work?)

    Rating system

    Ratings

    Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

    How to rate a module

    Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

    		(0 ratings)

Recently Viewed

This feature requires Javascript to be enabled.

The Cramer-Rao Lower Bound

Module by: Clayton Scott, Robert Nowak

Note: Your browser may not currently support MathML. See our browser support page for additional details. You can always view the correct math in the PDF version.

The Cramer-Rao Lower Bound (CRLB) sets a lower bound on the variance of any unbiased estimator. This can be extremely useful in several ways:

  1. If we find an estimator that achieves the CRLB, then we know that we have found an MVUB estimator!
  2. The CRLB can provide a benchmark against which we can compare the performance of any unbiased estimator. (We know we're doing very well if our estimator is "close" to the CRLB.)
  3. The CRLB enables us to rule-out impossible estimators. That is, we know that it is physically impossible to find an unbiased estimator that beats the CRLB. This is useful in feasibility studies.
  4. The theory behind the CRLB can tell us if an estimator exists that achieves the bound.

Estimator Accuracy

Consider the likelihood function px|θ p θ x , where θθ is a scalar unknown (parameter). We can plot the likelihood as a function of the unknown, as shown in Figure 1.

Figure 1
Figure 1 ()
The more "peaky" or "spiky" the likelihood function, the easier it is to determind the unknown parameter.

Example 1

Suppose we observe x=A+w x A w where w∼σσ2 w σ σ 2 and AA is an unknown parameter. The "smaller" the noise w w is, the easier it will be to estimate AA from the observation xx.

Suppose A=3 A 3 and σ=1/3 σ 13 .

Figure 2
Figure 2 ()
Given this density function, we can easily rule-out estimates of AA greater than 4 or less than 2, since it is very unlikely that such AA could give rise to out observation.

On the other hand, suppose σ=1 σ 1 .

Figure 3
Figure 3 ()
In this case, it is very difficult to estimate AA. Since the noise power is larger, it is very difficult to distinguish AA from the noise.

The key thing to notice is that the estimation accuracy of AA depends on σσ, which in effect determines the peakiness of the likelihood. The more peaky, the better localized the data is about the true parameter.

To quantify the notion, note that the peakiness is effectively measured by the negative of the second derivative of the log-likelihood at its peak, as seen in Figure 4.

Figure 4
Figure 4 ()

Example 2

x=A+w x A w

logpx|A=-log2πσ2−12σ2x−A2 p A x 2 σ 2 1 2 σ 2 x A 2 (1)
∂∂Alogpx|A=1σ2x−A A p A x 1 σ 2 x A
-∂2∂A2logpx|A=1σ2 A 2 p A x 1 σ 2 (2)
The curvature increases as σ2 σ 2 decreases (curvature=peakiness).

In general, the curavture will depend on the observation data; -∂2∂θ2logpx|A θ 2 p A x is a function of xx. Therefore, an average measure of curvature is more appropriate.

-E∂2∂θ2logpx|θ θ 2 p θ x (3)
This average-out randomness due to the data and is a function of θθ alone.

We are now ready to state the CRLB theorem.

Theorem 1: Cramer-Rao Lower Bound Theorem

Assume that the pdf px|θ p θ x satisfies the "regularity" condition ∀θ:E∂∂θlogpx|θ=0 θ θ p θ x 0 where the expectation is take with respect to px|θ p θ x . Then, the variance of any unbiased estimator θ ̂ θ must satisfy

σ θ ̂2≥1-E∂2∂θ2logpx|θ θ 1 θ 2 p θ x (4)
where the derivative is evaluated at the true value of θθ and the expectation is with respect to px|θ p θ x . Moreover, an unbiased estimator may be found that attains the bound for all θθ if and only if
∂∂θlogpx|θ=Iθgθ−θ θ p θ x I θ g θ θ (5)
for some functions gg and II.

The corresponding estimator is MVUB and is given by θ ̂=gx θ g x , and the minimum variance is 1Iθ 1 I θ .

Example

x=A+w x A w where w∼0σ2 w 0 σ 2 θ=A θ A ∀A:E∂∂θlogp=E1σ2x−A=0 A θ p 1 σ 2 x A 0 CRLB=1-E∂2∂θ2logp=11σ2=σ2 CRLB 1 θ 2 p 1 1 σ 2 σ 2 Therefore, any unbiased estimator A ̂ A has σ A ̂2≥σ2 A σ 2 . But we know that A ̂=x A x has σ A ̂2=σ2 A σ 2 . Therefore, A ̂=x A x is the MVUB estimator.

note:
θ=A θ A Iθ=1σ2 I θ 1 σ 2 gx=x g x x

Proof

First consider the reguarity condition: E∂∂θlogpx|θ=0 θ p θ x 0

note:
E∂∂θlogpx|θ=∫∂∂θlogpx|θpx|θdθ=∫∂∂θpx|θdθ θ p θ x θ θ p θ x p θ x θ θ p θ x
Now assuming that we can interchange order of differentiation and integration E∂∂θlogpx|θ=∂∂θ∫px|θdθ=∂∂θ1=0 θ p θ x θ θ p θ x θ 1 0 So the regularity condition is satisfied whenever this interchange is possible1; i.e., when derivative is well-defined, fails for uniform density.

Now lets derive the CRLB for a scalar parameter α=gθ α g θ , where the pdf is px|θ p θ x . Consider any unbiased estimator of α α: α ̂∈E α ̂=α=gθ α α α g θ Note that this is equivalent to ∫ α ̂px|θdx=gθ x α p θ x g θ where α ̂ α is unbiased. Now differentiate both side ∫ α ̂∂∂θpx|θdx=∂∂θgθ x α θ p θ x θ g θ or ∫ α ̂∂∂θlogpx|θpx|θdx=∂∂θgθ x α θ p θ x p θ x θ g θ

Now, exmploiting the regularity condition,

∫ α ̂−α∂∂θlogpx|θpx|θdx=∂∂θgθ x α α θ p θ x p θ x θ g θ (6)
since ∫α∂∂θlogpx|θpx|θdx=αElogpx|θ=0 x α θ p θ x p θ x α p θ x 0 Now apply the Cauchy-Schwarz inequality to the integral above: ∂∂θgθ2=∫ α ̂−α∂∂θlogpx|θpx|θdx2 θ g θ 2 x α α θ p θ x p θ x 2 ∂∂θgθ2≤∫ α ̂−α2px|θdx∫∂∂θlogpx|θpx|θdθ θ g θ 2 x α α 2 p θ x θ θ p θ x p θ x σ α ̂2 α is ∫ α ̂−α2px|θdx x α α 2 p θ x , so
σ α ̂2≥∂∂θgθ2E∂∂θlogpx|θ2 α θ g θ 2 θ p θ x 2 (7)
Now we note that E∂∂θlogpx|θ2=-E∂2∂θ2logpx|θ θ p θ x 2 θ 2 p θ x Why? Regularity condition. E∂∂θlogpx|θ=∫∂∂θlogpx|θpx|θdx=0 θ p θ x x θ p θ x p θ x 0 Thus, ∂∂θ∫∂∂θlogpx|θpx|θdx=0 θ x θ p θ x p θ x 0 or ∫∂2∂θ2logpx|θpx|θ+∂∂θlogpx|θ∂∂θpx|θdx=0 x θ 2 p θ x p θ x θ p θ x θ p θ x 0 Therefore, -E∂2∂θ2logpx|θ=∫∂∂θlogpx|θ∂∂θlogpx|θpx|θdx=E∂∂θlogpx|θ2 θ 2 p θ x x θ p θ x θ p θ x p θ x θ p θ x 2 Thus, Equation 7 becomes σ α ̂2≥∂∂θgθ2-E∂2∂θ2logpx|θ α θ g θ 2 θ 2 p θ x
note:
If gθ=θ g θ θ , then numerator is 1.

Example: DC Level in White Guassian Noise

∀n,n∈1…N: x n =A+ w n n n 1 … N x n A w n where w n ∼0σ2 w n 0 σ 2 px|A=12πσ2N2ⅇ-1σ2∑n=1N x n −A2 p A x 1 2 σ 2 N 2 1 σ 2 n 1 N x n A 2 ∂∂Alogpx|A=∂∂A-log2πσ2N2−12σ2∑n=1N x n −A2=1σ2∑n=1N x n −A A p A x A 2 σ 2 N 2 1 2 σ 2 n 1 N x n A 2 1 σ 2 n 1 N x n A E∂∂Alogpx|A=0 A p A x 0 ∂2∂A2logpx|A=-Nσ2 A 2 p A x N σ 2 Therefore, the variance of any unbiased estimator satisfies: σ A ̂2≥σ2N A σ 2 N The sample-mean estimator A ̂=1N∑n=1N x n A 1 N n 1 N x n attains this bound and therefore is MVUB.

Corollary 1

When the CRLB is attained σ θ ̂2=1Iθ θ 1 I θ where Iθ=-E∂2∂θ2logpx|θ I θ θ 2 p θ x The quantity Iθ I θ is called Fisher Information that xx contains about θθ.

Proof

By CRLB Theorem, σ θ ̂2=1-E∂2∂θ2logpx|θ θ 1 θ 2 p θ x and ∂∂θlogpx|θ=Iθ θ ̂−θ θ p θ x I θ θ θ This yields ∂2∂θ2logpx|θ=∂∂θIθ θ ̂−θ−Iθ θ 2 p θ x θ I θ θ θ I θ which in turn yields -E∂2∂θ2logpx|θ=Iθ θ 2 p θ x I θ So, σ θ ̂2=1Iθ θ 1 I θ

The CRLB is not always attained.

Example 3: Phase Estimation

∀n,n∈1…N: x n =Acos2π f 0 n+φ+ w n n n 1 … N x n A 2 f 0 n φ w n The amplitude and frequency are assumed known w n ∼0σ2 w n 0 σ 2 idd. px|φ=12πσ2N2ⅇ-12σ2∑n=1N x n −Acos2π f 0 n+φ p φ x 1 2 σ 2 N 2 1 2 σ 2 n 1 N x n A 2 f 0 n φ ∂∂φlogpx|φ=-Aσ2∑n=1N x n sin2π f 0 n+φ−A2sin4π f 0 n+φ φ p φ x A σ 2 n 1 N x n 2 f 0 n φ A 2 4 f 0 n φ ∂2∂φ2logpx|φ=-Aσ2∑n=1N x n cos2π f 0 n+φ−Acos2π f 0 n+2φ φ 2 p φ x A σ 2 n 1 N x n 2 f 0 n φ A 2 f 0 n 2 φ -E∂2∂φ2logpx|φ=A2σ2∑n=1N1/2+1/2cos4π f 0 n+2φ−cos4π f 0 n+2φ φ 2 p φ x A 2 σ 2 n 1 N 12 12 4 f 0 n 2 φ 4 f 0 n 2 φ Since Iφ=-E∂2∂φ2logpx|φ I φ φ 2 p φ x , Iφ≈NA22σ2 I φ N A 2 2 σ 2 because ∀ f 0 ,0< f 0 <k:1N∑cos4π f 0 n≈0 f 0 0 f 0 k 1 N 4 f 0 n 0 Therefore, σ φ ̂2≥2σ2NA2 φ 2 σ 2 N A 2

In this case, it can be shown that there does not exist a gg such that ∂∂φlogpx|φ≠Iφgx−φ φ p φ x I φ g x φ Therefore, an unbiased phase estimator that attains the CRLB does not exist.

However, a MVUB estimator may still exist--only its variance will be larger than the CRLB.

Efficiency

An estimator which is unbiased and attains the CRLB is said to be efficient.

Example 4

Sample-mean estimator is efficient.

Example 5

Supposed three unbiased estimators exist for a param θθ.

Figure 5
Figure 5 ()
Figure 6
Figure 6 ()

Example 6: Sinusoidal Frequency Estimation

∀ f 0 ,0< f 0 <1/2: s n f 0 =Acos2π f 0 n+φ f 0 0 f 0 12 s n f 0 A 2 f 0 n φ ∀n,n∈1…N: x n = s n f 0 + w n n n 1 … N x n s n f 0 w n AA and φφ are known, while f0 f0 is unknown. σf̂02≥σ2A2∑n=1N2πnsin2π f 0 n+φ2 f 0 σ 2 A 2 n 1 N 2 n 2 f 0 n φ 2 Suppose A2σ2=1 A 2 σ 2 1 (SNR), where N=10 N 10 and φ=0 φ 0 .

Figure 7: Some frequencies are easier to estimator (lower CRLB, but not necessarily just lower bound) than others.
Figure 7 ()

CRLB for Vector Parameter

θ= θ 1 θ 2 ⋮ θ p θ θ 1 θ 2 ⋮ θ p θ ̂ θ is unbiased, i.e., ∀i,i∈1…p:Eθ̂i= θ i i i 1 … p θ i θ i

CRLB

σθ̂i2≥Iθ-1ii θ i I θ i i where ∀i∧j:Iθij=-E∂2∂ θ i ∂ θ j logpx|θ i j I θ i j θ i θ j p θ x Iθ I θ is the Fisher Information Matrix.

Theorem 2: Cramer-Rao Lower Bound - Vector Parameter

Assume the pdf px|φ p φ x satisfies the "regularity" condition ∀θ:E∂∂θlogpx|θ=0 θ θ p θ x 0 Then the convariance matrix of any unbiased estimator θ ̂ θ satisfies C θ ^ −Iθ-1≥0 C θ ^ I θ 0 (meaning C θ ^ −Iθ-1 C θ ^ I θ is p.s.d.) The Fisher Information matrix is Iθij=-E∂2∂θ2logpx|θ I θ i j θ 2 p θ x Furthermore, θ ̂ θ attains the CRLB ( C θ ^ =Iθ-1 C θ ^ I θ ) iff ∂∂θlogpx|θ=Iθgx−θ θ p θ x I θ g x θ and θ ̂=gx θ g x

Example: DC Level in White Guassian Noise

∀n,n∈1…N: x n =A+ w n n n 1 … N x n A w n AA is unknown and w n ∼0σ2 w n 0 σ 2 , where σ2 σ 2 is unknown. θ=Aσ2 θ A σ 2 logpx|θ=-N2log2π−N2logσ2−12σ2∑n=2N x n −A2 p θ x N 2 2 N 2 σ 2 1 2 σ 2 n 2 N x n A 2 ∂∂Alogpx|θ=1σ2∑n=1N x n −A A p θ x 1 σ 2 n 1 N x n A ∂∂σ2logpx|θ=-Nσ2+12σ4∑n=1N x n −A2 σ 2 p θ x N σ 2 1 2 σ 4 n 1 N x n A 2 ∂2∂A2logpx|θ=-Nσ2→-Nσ2 A 2 p θ x N σ 2 N σ 2 ∂2∂A∂σ2logpx|θ=-1σ4∑n=1N x n −A→0 A σ 2 p θ x 1 σ 4 n 1 N x n A 0 ∂2∂σ22logpx|θ=N2σ4−1σ6∑n=1N x n −A2→-N2σ4 σ 2 2 p θ x N 2 σ 4 1 σ 6 n 1 N x n A 2 N 2 σ 4 Which leads to Iθ=Nσ200N2σ4 I θ N σ 2 0 0 N 2 σ 4 σ A ̂2≥σ2N A σ 2 N σ σ2 ̂2≥2σ4N σ 2 2 σ 4 N Note that the CRLB for A ̂ A is the same whether or not σ2 σ 2 is known. This happens in this case due to the diagonal nature of the Fisher Information Matrix.

In general the Fisher Information Matrix is not diagonal and consequently the CRLBs will depend on other unknown parameters.

Footnotes

  1. This is simply the Fundamental Theorem of Calculus applied to px|θ p θ x . So long as px|θ p θ x is absolutely continuous with respect to the Lebesgue measure, this is possible.

Glossary

idd:
independent and identically distributed

Comments, questions, feedback, criticisms?

Send feedback