Non-Random Parameters 1.4 2003/06/03 2003/08/27 13:47:41.604 GMT-5 Don Johnson dhj@rice.edu Don Johnson dhj@rice.edu Eileen Krause erkrause@rice.edu Kyle Clarkson kclarks@rice.edu Elizabeth Gregory lizzardg@rice.edu Kevin Duh kevinduh@rice.edu Mariyah Poonawala mariyah@rice.edu Matthew Jeanes mjeanes@rice.edu Jeffrey Silverman jsilv@rice.edu In those cases where a probability density for the parameters cannot be assigned, the model evaluation problem can be solved in several ways; the methods used depend on the form of the likelihood ratio and the way in which the parameter(s) enter the problem. In the Gaussian problem we have discussed so often, the threshold used in the likelihood ratio test η may be unity. In this case, examination of the resulting computations required reveals that implementing the test does not require knowledge of the variance of the observations (see this problem). Thus, if the common variance of the underlying Gaussian distributions is not known, this lack of knowledge has no effect on the optimum decision rule. This happy situation - knowledge of the value of a parameter is not required by the optimum decision rule - occurs rarely, but should be checked before using more complicated procedures. A second fortuitous situation occurs when the sufficient statistic as well as its probability density under one of the models do not depend on the unknown parameter(s). Although the sufficient statistic's threshold γ expressed in terms of the likelihood ratio's threshold η depends on the unknown parameters, γ may be computed as a single value using the Neyman-Pearson criterion if the computation of the false-alarm probability does not involve the unknown parameters. Continuing the example of the previous section, let's consider the situation where the value of the mean of each observation under model ℳ 1 is not known. The sufficient statistic is the sum of the observations (that quantity doesn't depend on m ) and the distribution of the observation vector under model ℳ 0 does not depend on m (allowing computation of the false-alarm probability). However, a subtlety emerges; in the derivation of the sufficient statistic, we had to divide by the value of the mean. The critical step occurs once the logarithm of the likelihood ratio is manipulated to obtain m l 0 L 1 r l ≷ ℳ 0 ℳ 1 σ 2 η L m 2 2 Recall that only positively monotonic transformations can be applied; if a negatively monotonic operation is applied to this inequality (such as multiplying both sides by -1), the inequality reverses. If the sign of m is known, it can be taken into account explicitly and a sufficient statistic results. If, however, the sign is not known, the above expression cannot be manipulated further and the left side constitutes the sufficient statistic for this problem. The sufficient statistic then depends on the unknown parameter and we cannot develop a decision rule in this case. If the sign is known, we can proceed. Assuming the sign of m is positive, the sufficient statistic is the sum of the observations and the threshold γ is found by γ L σ Q P F Note that if the variance σ 2 instead of the mean were unknown, we could not compute the threshold. The difficulty lies not with the sufficient statistic (it doesn't depend on the variance), but with the false alarm probability as the expression indicates. Another approach is required to deal with the unknown-variance problem. When this situation occurs - the sufficient statistic and the false-alarm probability can be computed without needing the parameter in question, we have established what is known as a uniformly most powerful test (or UMP test) (Cramér; p.529-531), (van Trees; p.89ff). If an UMP test does not exist, which can only be demonstrated by explicitly finding the sufficient statistic and evaluating its probability distribution, then the composite hypothesis testing problem cannot be solved without some value for the parameter being used. This seemingly impossible situation - we need the value of the parameter that is assumed unknown - can be approached by noting that some data is available for "guessing" the value of the parameter. If a reasonable guess could be obtained, it could then be used in our model evaluation procedures developed in this chapter. The data available for estimating unknown parameters are precisely the data used in the decision rule. Procedures intended to yield "good" guesses of the value of a parameter are said to be parameter estimates. Estimation procedures are the topic of the next chapter; there we will explore a variety of estimation techniques and develop measure of estimate quality. For the moment, these issues are secondary; even if we knew the size of the estimation error, for example, the more pertinent issue is how the imprecise parameter value affects the performance probabilities. We can compute these probabilities without explicitly determining the estimate's error characteristics. One parameter estimation procedure that fits nicely into the composite hypothesis testing problem is the maximum likelihood estimate. The maximum likelihood estimation procedure and its characteristics are fully described in this section. Letting r denote the vector of observables and θ a vector of parameters, the maximum likelihood estimate of θ , θ ML , is that value of θ that maximizes the conditional density p r θ r of the observations given the parameter values. To use θ ML in our decision rule, we estimate the parameter vector separately for each model, use the estimated value in the conditional density of the observations, and compute the likelihood ratio. This procedure is termed the generalized likelihood ratio test for the unknown parameter problem in hypothesis testing (Lehmann; p.16), (van Trees; p.92ff). Λ r θ p r ℳ 1 θ r θ p r ℳ 0 θ r Note that we do not find that value of the parameter that (necessarily) maximizes the likelihood ratio. Rather, we estimate the parameter value most consistent with the observed data in the context of each assumed model (hypothesis) of data generation. In this way, the estimate conforms with each potential model rather than being determined by some amalgam of supposedly mutually exclusive models. Returning to our Gaussian example, assume that the variance σ 2 is known but that the mean under ℳ 1 is unknown. ℳ 0 :   r 0 σ 2 I ℳ 1 :   r m σ 2 I m m … m ,   m ? The unknown quantity occurs only in the exponent of the conditional density under ℳ 1 ; to maximize this density, we need only to maximize the exponent. Thus, we consider the derivative of the exponent with respect to m . m m ML m 1 2 σ 2 l 0 L 1 r l m 2 0 l 0 L 1 r l m ML 0 The solution of this equation is the average value of the observations m ML 1 L l 0 L 1 r l To derive the decision rule, we substitute this estimate in the conditional density for ℳ 1 . The critical term, the exponent of this density, is manipulated to obtain 1 2 σ 2 l 0 L 1 r l 1 L k 0 L 1 r k 2 1 2 σ 2 l 0 L 1 r l 2 1 L l 0 L 1 r l 2 Noting that the first term in this exponent is identical to the exponent of the denominator in the likelihood ratio, the generalized likelihood ratio becomes Λ r 1 2 L σ 2 l 0 L 1 r l 2 The sufficient statistic thus becomes the square (or equivalently the magnitude) of the summed observations. Compare this result with that obtained in . There, an UMP test existed if we knew the sign of m and the sufficient statistic was the sum of the observations. Here, where we employed the generalized likelihood ratio test, we made no such assumptions about m ; this generality accounts for the difference in sufficient statistic. Which test do you think would lead to a greater detection probability for a given false-alarm probability? Once the generalized likelihood ratio is determined, we need to determine the threshold. If the a priori probabilities π 0 and π 1 are known, the evaluation of the threshold proceeds in the usual way. If they are not known, all of the conditional densities must not depend on the unknown parameters lest the performance probabilities also depend upon them. In most cases, the original model evaluation problem is posed in such a way that one of the models does not depend on the unknown parameter; a criterion on the performance probability related to that model can then be established via the Neyman-Pearson procedure. If not the case, the threshold cannot be computed and the threshold must be set experimentally: we force one of the models to be true and modify the threshold on the sufficient statistic until the desired level of performance is reached. Despite this non-mathematical approach, the overall performance of the model evaluation procedure will be optimum because of the results surrounding the Neyman-Pearson criterion.

The two-model testing problem can be abstractly described as a communication channel where the inputs are the models and the outputs are the decisions. The transition probabilities are related to the false-alarm ( P F ) and detection ( P D ) probabilities.

H. Cramér Mathematical Methods of Statistics Princeton University Press 1946 Princeton, NJ H.L. van Trees Detection, Estimation, and Modulation Theory, Part I John Wiley and Sons 1968 New York E.L. Lehmann Testing Statistical Hypotheses John Wiley and Sons 1986 New York second edition