**Likelihood function**

From a statistical standpoint, the data vector

Let denote the *probability distribution function * (PDF) by *y* given the parameter *w*. The parameter vector

To illustrate the idea of a PDF, consider the simplest case with one observation and one parameter, that is, *x* represents the number of successes in a sequence of 10 independent binary trials (e.g., coin tossing experiment) and that the probability of a success on any one trial, represented by the parameter,

which is known as the binomial probability distribution. The shape of this PDF is shown in the top panel of Figure 1. If the parameter value is changed to say *w* = 0.7, a new PDF is obtained as *n*:

which as a function of *y* specifies the probability of data *y* for a given value of the parameter

**Maximum Likelihood Estimation**

Once data have been collected and the likelihood function of a model given the data is determined, one is in a position to make statistical inferences about the population, that is, the probability distribution that underlies the data. Given that different parameter values index different probability distributions (Figure 1), we are interested in finding the parameter value that corresponds to the desired PDF.

The principle of maximum likelihood estimation (MLE), originally developed by R. A. Fisher in the 1920s, states that the desired probability distribution be the one that makes the observed data most likely, which is obtained by seeking the value of the parameter vector that maximizes the likelihood function

Let *p* equal the probability of success in a sequence of Bernoulli trials or the proportion of the large population with a certain characteristic. The method of moments estimate for *p* is relative frequency of success (having that characteristic). It will be shown below that the maximum likelihood estimate for *p* is also the relative frequency of success.

Suppose that *X* is *X* is *p*, where

which is the joint p.d.f. of *p* is to regard this probability (or joint p.d.f.) as a function of *p* and find the value of *p* that maximizes it. That is, find the *p* value most likely to have produced these sample values. The joint p.d.f., when regarded as a function of *p*, is frequently called the likelihood function. Thus here the likelihood function is:

To find the value of *p* that maximizes

Setting this first derivative equal to zero gives

Since

The corresponding statistics, namely

When finding a maximum likelihood estimator, it is often easier to find the value of parameter that minimizes the natural logarithm of the likelihood function rather than the value of the parameter that minimizes the likelihood function itself. Because the natural logarithm function is an increasing function, the solution will be the same. To see this, the example which was considered above gives for

To find the maximum, set the first derivative equal to zero to obtain

which is the same as previous equation. Thus the solution is *p* is

Motivated by the preceding illustration, the formal definition of maximum likelihood estimators is presented. This definition is used in both the discrete and continuous cases.
In many practical cases, these estimators (and estimates) are unique. For many applications there is just one unknown parameter. In this case the likelihood function is given by