Consider the likelihood function
px|θ
p
θ
x
, where θθ is a
scalar unknown (parameter). We can plot the likelihood as a
function of the unknown, as shown in Figure 1.
The more "peaky" or "spiky" the likelihood function, the
easier it is to determind the unknown parameter.
Suppose we observe
x=A+w
x
A
w
where
w∼σσ2
w
σ
σ
2
and AA is an unknown
parameter. The "smaller" the noise w
w is, the easier it will be to estimate
AA from the observation
xx.
Suppose
A=3
A
3
and
σ=1/3
σ
13
.
Given this density function, we can easily rule-out
estimates of
AA greater than 4
or less than 2, since it is very unlikely that such
AA could give rise to out
observation.
On the other hand, suppose
σ=1
σ
1
.
In this case, it is very difficult to estimate
AA. Since the noise power is
larger, it is very difficult to distinguish
AA from the noise.
The key thing to notice is that the estimation accuracy of
AA depends on
σσ, which in effect
determines the peakiness of the likelihood. The more peaky,
the better localized the data is about the true parameter.
To quantify the notion, note that the peakiness is
effectively measured by the negative of the second
derivative of the log-likelihood at its peak, as seen in
Figure 4.
x=A+w
x
A
w
logpx|A=-log2πσ2−12σ2x−A2
p
A
x
2
σ
2
1
2
σ
2
x
A
2
(1)
∂∂Alogpx|A=1σ2x−A
A
p
A
x
1
σ
2
x
A
-∂2∂A2logpx|A=1σ2
A
2
p
A
x
1
σ
2
(2)
The curvature increases as
σ2
σ
2
decreases (curvature=peakiness).
In general, the curavture will depend on the observation data;
-∂2∂θ2logpx|A
θ
2
p
A
x
is a function of xx.
Therefore, an average measure of curvature is more appropriate.
-E∂2∂θ2logpx|θ
θ
2
p
θ
x
(3)
This average-out randomness due to the data and is a
function of
θθ alone.
We are now ready to state the CRLB theorem.
Assume that the pdf
px|θ
p
θ
x
satisfies the "regularity" condition
∀θ:E∂∂θlogpx|θ=0
θ
θ
p
θ
x
0
where the expectation is take with respect to
px|θ
p
θ
x
. Then, the variance of any unbiased estimator
θ
̂
θ
must satisfy
σ
θ
̂2≥1-E∂2∂θ2logpx|θ
θ
1
θ
2
p
θ
x
(4)
where the derivative is evaluated at the
true value of
θθ
and the expectation is with respect to
px|θ
p
θ
x
. Moreover, an unbiased estimator may be
found that attains the bound for all
θθ if and only if
∂∂θlogpx|θ=Iθgθ−θ
θ
p
θ
x
I
θ
g
θ
θ
(5)
for some functions
gg and
II.
The corresponding estimator is MVUB and is given by
θ
̂=gx
θ
g
x
, and the minimum variance is
1Iθ
1
I
θ
.
x=A+w
x
A
w
where
w∼0σ2
w
0
σ
2
θ=A
θ
A
∀A:E∂∂θlogp=E1σ2x−A=0
A
θ
p
1
σ
2
x
A
0
CRLB=1-E∂2∂θ2logp=11σ2=σ2
CRLB
1
θ
2
p
1
1
σ
2
σ
2
Therefore, any unbiased estimator
A
̂
A
has
σ
A
̂2≥σ2
A
σ
2
. But we know that
A
̂=x
A
x
has
σ
A
̂2=σ2
A
σ
2
. Therefore,
A
̂=x
A
x
is the MVUB estimator.
θ=A
θ
A
Iθ=1σ2
I
θ
1
σ
2
gx=x
g
x
x
First consider the reguarity condition:
E∂∂θlogpx|θ=0
θ
p
θ
x
0
E∂∂θlogpx|θ=∫∂∂θlogpx|θpx|θdθ=∫∂∂θpx|θdθ
θ
p
θ
x
θ
θ
p
θ
x
p
θ
x
θ
θ
p
θ
x
Now assuming that we can interchange order of differentiation and
integration
E∂∂θlogpx|θ=∂∂θ∫px|θdθ=∂∂θ1=0
θ
p
θ
x
θ
θ
p
θ
x
θ
1
0
So the regularity condition is satisfied whenever this
interchange is possible;
i.e., when
derivative is well-defined, fails for uniform
density.
Now lets derive the CRLB for a scalar parameter
α=gθ
α
g
θ
, where the pdf is
px|θ
p
θ
x
. Consider any unbiased estimator of
α α:
α
̂∈E
α
̂=α=gθ
α
α
α
g
θ
Note that this is equivalent to
∫
α
̂px|θdx=gθ
x
α
p
θ
x
g
θ
where
α
̂
α
is unbiased. Now differentiate both side
∫
α
̂∂∂θpx|θdx=∂∂θgθ
x
α
θ
p
θ
x
θ
g
θ
or
∫
α
̂∂∂θlogpx|θpx|θdx=∂∂θgθ
x
α
θ
p
θ
x
p
θ
x
θ
g
θ
Now, exmploiting the regularity condition,
∫
α
̂−α∂∂θlogpx|θpx|θdx=∂∂θgθ
x
α
α
θ
p
θ
x
p
θ
x
θ
g
θ
(6)
since
∫α∂∂θlogpx|θpx|θdx=αElogpx|θ=0
x
α
θ
p
θ
x
p
θ
x
α
p
θ
x
0
Now apply the
Cauchy-Schwarz
inequality to the
integral above:
∂∂θgθ2=∫
α
̂−α∂∂θlogpx|θpx|θdx2
θ
g
θ
2
x
α
α
θ
p
θ
x
p
θ
x
2
∂∂θgθ2≤∫
α
̂−α2px|θdx∫∂∂θlogpx|θpx|θdθ
θ
g
θ
2
x
α
α
2
p
θ
x
θ
θ
p
θ
x
p
θ
x
σ
α
̂2
α
is
∫
α
̂−α2px|θdx
x
α
α
2
p
θ
x
, so
σ
α
̂2≥∂∂θgθ2E∂∂θlogpx|θ2
α
θ
g
θ
2
θ
p
θ
x
2
(7)
Now we note that
E∂∂θlogpx|θ2=-E∂2∂θ2logpx|θ
θ
p
θ
x
2
θ
2
p
θ
x
Why? Regularity condition.
E∂∂θlogpx|θ=∫∂∂θlogpx|θpx|θdx=0
θ
p
θ
x
x
θ
p
θ
x
p
θ
x
0
Thus,
∂∂θ∫∂∂θlogpx|θpx|θdx=0
θ
x
θ
p
θ
x
p
θ
x
0
or
∫∂2∂θ2logpx|θpx|θ+∂∂θlogpx|θ∂∂θpx|θdx=0
x
θ
2
p
θ
x
p
θ
x
θ
p
θ
x
θ
p
θ
x
0
Therefore,
-E∂2∂θ2logpx|θ=∫∂∂θlogpx|θ∂∂θlogpx|θpx|θdx=E∂∂θlogpx|θ2
θ
2
p
θ
x
x
θ
p
θ
x
θ
p
θ
x
p
θ
x
θ
p
θ
x
2
Thus,
Equation 7 becomes
σ
α
̂2≥∂∂θgθ2-E∂2∂θ2logpx|θ
α
θ
g
θ
2
θ
2
p
θ
x
If
gθ=θ
g
θ
θ
, then numerator is 1.
∀n,n∈1…N:
x
n
=A+
w
n
n
n
1
…
N
x
n
A
w
n
where
w
n
∼0σ2
w
n
0
σ
2
px|A=12πσ2N2ⅇ-1σ2∑n=1N
x
n
−A2
p
A
x
1
2
σ
2
N
2
1
σ
2
n
1
N
x
n
A
2
∂∂Alogpx|A=∂∂A-log2πσ2N2−12σ2∑n=1N
x
n
−A2=1σ2∑n=1N
x
n
−A
A
p
A
x
A
2
σ
2
N
2
1
2
σ
2
n
1
N
x
n
A
2
1
σ
2
n
1
N
x
n
A
E∂∂Alogpx|A=0
A
p
A
x
0
∂2∂A2logpx|A=-Nσ2
A
2
p
A
x
N
σ
2
Therefore, the variance of any unbiased estimator satisfies:
σ
A
̂2≥σ2N
A
σ
2
N
The sample-mean estimator
A
̂=1N∑n=1N
x
n
A
1
N
n
1
N
x
n
attains this bound and therefore is MVUB.
When the CRLB is attained
σ
θ
̂2=1Iθ
θ
1
I
θ
where
Iθ=-E∂2∂θ2logpx|θ
I
θ
θ
2
p
θ
x
The quantity
Iθ
I
θ
is called Fisher Information that
xx contains about
θθ.
By CRLB Theorem,
σ
θ
̂2=1-E∂2∂θ2logpx|θ
θ
1
θ
2
p
θ
x
and
∂∂θlogpx|θ=Iθ
θ
̂−θ
θ
p
θ
x
I
θ
θ
θ
This yields
∂2∂θ2logpx|θ=∂∂θIθ
θ
̂−θ−Iθ
θ
2
p
θ
x
θ
I
θ
θ
θ
I
θ
which in turn yields
-E∂2∂θ2logpx|θ=Iθ
θ
2
p
θ
x
I
θ
So,
σ
θ
̂2=1Iθ
θ
1
I
θ
The CRLB is not always attained.
∀n,n∈1…N:
x
n
=Acos2π
f
0
n+φ+
w
n
n
n
1
…
N
x
n
A
2
f
0
n
φ
w
n
The amplitude and frequency are assumed known
w
n
∼0σ2
w
n
0
σ
2
idd.
px|φ=12πσ2N2ⅇ-12σ2∑n=1N
x
n
−Acos2π
f
0
n+φ
p
φ
x
1
2
σ
2
N
2
1
2
σ
2
n
1
N
x
n
A
2
f
0
n
φ
∂∂φlogpx|φ=-Aσ2∑n=1N
x
n
sin2π
f
0
n+φ−A2sin4π
f
0
n+φ
φ
p
φ
x
A
σ
2
n
1
N
x
n
2
f
0
n
φ
A
2
4
f
0
n
φ
∂2∂φ2logpx|φ=-Aσ2∑n=1N
x
n
cos2π
f
0
n+φ−Acos2π
f
0
n+2φ
φ
2
p
φ
x
A
σ
2
n
1
N
x
n
2
f
0
n
φ
A
2
f
0
n
2
φ
-E∂2∂φ2logpx|φ=A2σ2∑n=1N1/2+1/2cos4π
f
0
n+2φ−cos4π
f
0
n+2φ
φ
2
p
φ
x
A
2
σ
2
n
1
N
12
12
4
f
0
n
2
φ
4
f
0
n
2
φ
Since
Iφ=-E∂2∂φ2logpx|φ
I
φ
φ
2
p
φ
x
,
Iφ≈NA22σ2
I
φ
N
A
2
2
σ
2
because
∀
f
0
,0<
f
0
<k:1N∑cos4π
f
0
n≈0
f
0
0
f
0
k
1
N
4
f
0
n
0
Therefore,
σ
φ
̂2≥2σ2NA2
φ
2
σ
2
N
A
2
In this case, it can be shown that there does not exist a
gg such that
∂∂φlogpx|φ≠Iφgx−φ
φ
p
φ
x
I
φ
g
x
φ
Therefore, an unbiased phase estimator that attains the CRLB
does not exist.
However, a MVUB estimator may still exist--only its
variance will be larger than the CRLB.
An estimator which is unbiased and attains the CRLB is said to
be efficient.
Sample-mean estimator is efficient.
Supposed three unbiased estimators exist for a param
θθ.
∀
f
0
,0<
f
0
<1/2:
s
n
f
0
=Acos2π
f
0
n+φ
f
0
0
f
0
12
s
n
f
0
A
2
f
0
n
φ
∀n,n∈1…N:
x
n
=
s
n
f
0
+
w
n
n
n
1
…
N
x
n
s
n
f
0
w
n
AA and
φφ are known, while
f0
f0
is unknown.
σf̂02≥σ2A2∑n=1N2πnsin2π
f
0
n+φ2
f
0
σ
2
A
2
n
1
N
2
n
2
f
0
n
φ
2
Suppose
A2σ2=1
A
2
σ
2
1
(SNR), where
N=10
N
10
and
φ=0
φ
0
.