Maximum Likelihood Estimators of Parameters 1.4 2003/05/20 2003/08/18 17:05:14.804 GMT-5 Don Johnson dhj@rice.edu Don Johnson dhj@rice.edu Eileen Krause erkrause@rice.edu Kyle Clarkson kclarks@rice.edu Elizabeth Gregory lizzardg@rice.edu Kevin Duh kevinduh@rice.edu Mariyah Poonawala mariyah@rice.edu Matthew Jeanes mjeanes@rice.edu Jeffrey Silverman jsilv@rice.edu When the a priori density of a parameter is not known or the parameter itself is inconveniently described as a random variable, techniques must be developed that make no presumption about the relative possibilities of parameter values. Lacking this knowledge, we can expect the error characteristics of the resulting estimates to be worse than those which can use it. The maximum likelihood estimate θ ML r of a nonrandom parameter is, simply, that value which maximizes the likelihood function (the a priori density of the observations). Assuming that the maximum can be found by evaluating a derivative, θ ML r is defined by θ θ ML θ p θ r 0 The logarithm of the likelihood function may also be used in this maximization. Let r l be a sequence of independent, identically distributed Gaussian random variables having an unknown mean θ but a known variance σ n 2 . Often, we cannot assign a probability density to a parameter of a random variable's density; we simply do not know what the parameter's value is. Maximum likelihood estimates are often used in such problems. In the specific case here, the derivative of the logarithm of the likelihood function equals θ p θ r 1 σ n 2 l 0 L 1 r l θ The solution of this equation is the maximum likelihood estimate, which equals the sample average. θ ML 1 L l 0 L 1 r l The expected value of this estimate θ θ ML equals the actual value θ, showing that the maximum likelihood estimate is unbiased. The mean-squared error equals σ n 2 L and we infer that this estimate is consistent.

Parameter Vectors The maximum likelihood procedure (as well as the others being discussed) can be easily generalized to situations where more than one parameter must be estimated. Letting θ denote the parameter vector, the likelihood function is now expressed as p θ r . The maximum likelihood estimate θ ML of the parameter vector is given by the location of the maximum of the likelihood function (or equivalently of its logarithm). Using derivatives, the calculation of the maximum likelihood estimate becomes θ θ ML θ p θ r 0 where ∇ θ denotes the gradient with respect to the parameter vector. This equation means that we must estimate all of the parameter simultaneously by setting the partial of the likelihood function with respect to each parameter to zero. Given P parameters, we must solve in most cases a set of P nonlinear, simultaneous equations to find the maximum likelihood estimates.

Let's extend the previous example to the situation where neither the mean nor the variance of a sequence of independent Gaussian random variables is known. The likelihood function is, in this case, p θ r l 0 L 1 1 2 θ 2 1 2 θ 2 r l σ 1 2 Evaluating the partial derivatives of the logarithm of this quantity, we find the following set of two equations to solve for θ 1 , representing the mean, and θ 2 , representing the variance.The variance rather than the standard deviation is represented by θ 2 . The mathematics is messier and the estimator has less attractive properties in the latter case. This problem illustrates this point. 1 θ 2 l 0 L 1 r l θ 1 0 L 2 θ 2 1 2 θ 2 2 l 0 L 1 r l θ 1 2 0 The solution of this set of equations is easily found to be θ 1 ML 1 L l 0 L 1 r l θ 2 ML 1 L l 0 L 1 r l θ 1 ML 2 The expected value of θ 1 ML equals the actual value of θ 1 ; thus, this estimate is unbiased. However, the expected value of the estimate of the variance equals θ 2 L 1 L . The estimate of the variance is biased, but asymptotically unbiased. This bias can be removed by replacing the normalization of L in the averaging computation for θ 2 ML by L 1 .