Estimation Theory: Problems 1.5 2003/05/15 2003/08/20 17:37:51.125 GMT-5 Don Johnson dhj@rice.edu Don Johnson dhj@rice.edu Eileen Krause erkrause@rice.edu Kyle Clarkson kclarks@rice.edu Elizabeth Gregory lizzardg@rice.edu Kevin Duh kevinduh@rice.edu Mariyah Poonawala mariyah@rice.edu Matthew Jeanes mjeanes@rice.edu Jeffrey Silverman jsilv@rice.edu Estimates for identical parameters are heavily dependent on the assumed underlying probability densities. To understand this sensitivity better, consider the following variety of problems, each of which asks for estimates of quantities related to variance. Determine the bias and consistency in each case.

Compute the maximum a posteriori and maximum likelihood estimates of θ based on L statistically independent observations of a Maxwellian random variable r . r θ r 0 θ 0 p r θ r 2 π θ -32 r 2 1 2 r 2 θ θ θ 0 p θ θ λ λ θ

Find the maximum a posteriori estimate of the variance σ 2 from L statistically independent observations having the exponential density r r 0 p r r 1 σ 2 r σ 2 where the variance is uniformly distributed over the interval 0 σ max 2 .

Find the maximum likelihood estimate of the variance of L identically distributed, but dependent Gaussian random variables. Here, the covariance matrix is written K r σ 2 K ∼ r , where the normalized covariance matrix has trace tr K ∼ r L

Imagine yourself idly standing on the corner in a large city when you note the serial number of a passing beer truck. Because you are idle, you wish to estimate (guess may be more accurate here) how many beer trucks the city has from this single operation

Making appropriate assumptions, the beer truck's number is drawn from a uniform probability density ranging between zero and some unknown upper limit, find the maximum likelihood estimate of the upper limit.

Show that this estimate is biased.

In one of your extraordinarily idle moments, you observe throughout the city L beer trucks. Assuming them to be independent observations, now what is the maximum likelihood estimate of the total?

Is this estimate of θ biased? asymptotically biased? consistent?

We make L observations r 1 , … , r L of a parameter θ corrupted by additive noise ( r l θ n l ). The parameter θ is a Gaussian random variable [ θ 0 σ θ 2 ] and n l are statistically independent Gaussian random variables [ n l 0 σ n 2 ].

Find the MMSE estimate of θ .

Find the maximum a posteriori estimate of θ .

Compute the resulting mean-squared error for each estimate.

Consider an alternate procedure based on the same observations r l . Using the MMSE criterion, we estimate θ immediately after each observation. This procedure yields the sequence of estimates θ 1 r 1 , θ 2 r 1 r 2 , …, θ L r 1 … r L . Express θ 1 as a function of θ l - 1 , σ l - 1 2 , and r l . Here, σ l 2 denotes the variance of the estimation error of the l th estimate. Show that 1 σ l 2 1 σ θ 2 1 σ n 2

Although the maximum likelihood estimation procedure was not clearly defined until early in the 20th century, Gauss showed in 1905 that the Gaussian density It wasn't called the Gaussian density in 1805; this result is one of the reasons why it is. was the sole density for which the maximum likelihood estimate of the mean equaled the sample average. Let r 0 … r L - 1 be a sequence of statistically independent, identically distributed random variables.

What equation defines the maximum likelihood estimate m ML of the mean m when the common probability density function of the data has the form p r m ?

The sample average is, of course, l l r l L . Show that it minimizes the mean-square error l l r l m 2 .

Equating the sample average to m ML , combine this equation with the maximum likelihood equation to show that the Gaussian density uniquely satisfies the equations.

Because both equations equal 0, they can be equated. Use the fact that they must hold for all L to derive the result. Gauss thus showed that mean-squared error and the Gaussian density were closely linked, presaging ideas from modern robust estimation theory. In this example, we derived the maximum likelihood estimate of the mean and variance of a Gaussian random vector. You might wonder why we chose to estimate the variance σ 2 rather than the standard deviation σ . Using the same assumptions provided in the example, let's explore the consequences of estimating a function of a parameter (van Trees: Probs 2.4.9, 2.4.10).

Assuming that the mean is known, find the maximum likelihood estimates of first the variance, then the standard deviation.

Are these estimates biased?

Describe how these two estimates are related. Assuming that f · is a monotonic function, how are θ ML and f ( θ ) ML related in general? These results suggest a general question. Consider the problem of estimating some function of a parameter θ , say f 1 θ . The observed quantity is r and the conditional density p r θ r is known. Assume that θ is a nonrandom parameter.

What are the conditions for an efficient estimate f 1 θ to exist?

What is the lower bound on the variance of the error of any unbiased estimate of f 1 θ ?

Assume an efficient estimate of f 1 θ exists; when can an efficient estimate of some other function f 2 θ exist?

Let the observations r l consist of statistically independent, identically distributed Gaussian random variables having zero mean but unknown variance. We wish to estimate σ 2 , their variance.

Find the maximum likelihood estimate σ ML 2 and compute the resulting mean-squared error.

Show that this estimate is efficient.

Consider a new estimate σ NEW 2 given by σ NEW 2 α σ ML 2 , where α is a constant. Find the value of α that minimizes the mean-squared error for σ NEW 2 . Show that the mean-squared error of σ NEW 2 is less than that of σ ML 2 . Is this result compatible with this previous part?

Let the observations be of the form r H θ n where θ and n are statistically independent Gaussian random vectors. θ 0 K θ n 0 K n The vector θ has dimension M ; the vectors r and n have dimension N .

Derive the minimum mean-squared error estimate of θ , θ MMSE , from the relationship θ MMSE r θ

Show that this estimate and the optimum linear estimate θ LIN derived by the Orthogonality Principle are equal.

Find an expression for the mean-squared error when these estimates are used.

To illustrate the power of importance sampling, let's consider a somewhat naïve example. Let r have a zero-mean Laplacian distribution; we want to employ importance sampling techniques to estimate r γ (despite the fact that we can calculate it easily). Let the density for r ˜ be Laplacian having mean γ .

Find the weight c l that must be applied to each decision based on the variable r ˜ .

Find the importance sampling gain. Show that this gain means that a fixed number of simulations are needed to achieve a given percentage estimation error (as defined by the coefficient of variation). Express this number as a function of the criterion value for the coefficient of variation.

Now assume that the density for r ˜ is Laplacian, but with mean m . Optimize m by finding the value that maximizes the importance sampling gain.

Suppose we consider an estimate of the parameter θ having the form θ ℒ r C , where r denotes the vector of the observables and ℒ · is a linear operator. The quantity C is a constant. This estimate is not a linear function of the observables unless C 0 . We are interested in finding applications for which it is advantageous to allow C 0 . Estimates of this form we term "quasi-linear".

Show that the optimum (minimum mean-squared error) quasi-linear estimate satisfies ℒ ⋄ r C ⋄ θ ℒ r C 0 for all ℒ · and C where θ QLIN ℒ ⋄ r C ⋄ .

Find a general expression for the mean-squared error incurred by the optimum quasi-linear estimate.

Such estimates yield a smaller mean-squared error when the parameter θ has a nonzero mean. Let θ be a scalar parameter with mean m . The observables comprise a vector r having components given by r l θ n l , l 1 … N where n l are statistically independent Gaussian random variables [ n l 0 σ n 2 ] independent of θ . Compute expressions for θ QLIN and θ LIN . Verify that θ QLIN yields a smaller mean-squared error when m 0 .

In this section, we questioned the existence of an efficient estimator for signal parameters. We found in the succeeding example that an unbiased efficient estimator exists for the signal amplitude. Can a nonlinearly represented parameter, such as time delay, have an efficient estimator?

Simplify the condition for the existence of an efficient estimator by assuming it to be unbiased. Note carefully the dimensions of the matrices involved.

Show that the only solution in this case occurs when the signal depends "linearly" on the parameter vector.

In Poission problems, the number of events n occurring in the interval 0 T is governed by the probability distribution (see The Poission Process) n λ T n n λ T where λ is the average rate at which events occur.

What is the maximum likelihood estimate of average rate?

Does this estimate satisfy the Cramér-Rao bound?

In the "classic" radar problem, not only is the time of arrival of the radar pulse unknown but also the amplitude. In this problem, we seek methods of simultaneously estimating these parameters. The received signal r l is of the form r l θ 1 s l θ 2 n l where θ 1 is Gaussian with zero mean and variance σ 1 2 and θ 2 is uniformly distributed over the observation interval. Find the receiver that computes the maximum a posteriori estimates of θ 1 and θ 2 jointly. Draw a block diagram of this receiver and interpret its structure. We state without derivation the Cramér-Rao bound for estimates of signal delay (see this equation).

The parameter θ is the delay of the signal s · observed in additive, white Gaussian noise: r l s l θ n l , l 0 … L 1 . Derive the Cramér-Rao bound for this problem.

In Time-delay Estimation, this bound is claimed to be given by σ n 2 E β 2 , where β 2 is the mean-squared bandwidth. Derive this result from your general formula. Does the bound make sense for all values of signal-to-noise ratio E σ n 2 ?

Using optimal detection theory, derive the expression (see Time-Delay Estimation) for the probability of error incurred when trying to distinguish between a delay of τ and a delay of τ Δ . Consistent with the problem pposed for the Cramér-Rao bound, assume the delayed signals are observed in additive, white Gaussian noise.

In formulating detection problems, the signal as well as the noise are sometimes modeled as Gaussian processes. Let's explore what differences arise in the Cramér-Rao bound derived when the signal is deterministic. Assume that the signal contains unknown parameters θ , that it is statistically independent of the noise, and that the noise covariance matrix is known.

What forms do the conditional densities of the observations take under the two assumptions? What are the two covariance matrices?

Assuming the stochastic signal model, show that each element of the Fisher information matrix has the form F i j 1 2 tr K θ i K K θ j K where K denotes the covariance matrix of the observations. Make this expression more complex by assuming the noise complement has no unknown parameters.

Compare the stochastic and deterministic bounds, the latter is given by this equation, when the unknown signal parameters are amplitude and delay. Assume the noise covariance matrix equals σ n 2 I . Do these bounds have similar dependence on signal-to-noise ratio?

The histogram probability density estimator is a special case of a more general class of estimators known as kernel estimators. p r x 1 L l 0 L 1 k x r l Here, the kernel k · is usually taken to be a density itself.

What is the kernel for the histogram estimator.

Interpret the kernel estimator in signal processing terminology. Predict what the most time consuming computation of this estimate might be. Why?

Show that the sample average equals the expected value of a random variable having the density p r x regardless of the choice of kernel.

Random variables can be generated quite easily if the probability distribution function is "nice." Let X be a random variable having distribution function P X · .

Show that the random variable U P X X is uniformly distributed over 0 1 .

Based on this result, how would you generate a random variable having a specific density with a uniform random variable generator, which is commonly supplied with most computer and calculator systems?

How would you generate random variables having the hyperbolic secant density p X x 1 2 x 2 ?

Why is the Gaussian not in the class of "nice" probability distribution functions? Despite this fact, the Gaussian and other similarly unfriendly random variables can be generated using tabulated rather than analytic forms for the distribution function.

H.L. van Trees Detection, Estimation, and Modulation Theory, Part I John Wiley and Sons 1968 New York