Skip to content Skip to navigation

Connexions

You are here: Home » Content » Maximum Likelihood Estimation (MLE)

Navigation

Recently Viewed

This feature requires Javascript to be enabled.

Maximum Likelihood Estimation (MLE)

Module by: Ewa Paszek. E-mail the author

User rating (How does the rating system work?)
Ratings

Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

How to rate a module

Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

:
(0 ratings)

Summary: This course is a short series of lectures on Introductory Statistics. Topics covered are listed in the Table of Contents. The notes were prepared by Ewa Paszek and Marek Kimmel. The development of this course has been supported by NSF 0203396 grant.

Note: Your browser may not currently support MathML. See our browser support page for additional details. You can always view the correct math in the PDF version.

MAXIMUM LIKELIHOOD ESTIMATION (MLE)

Likelihood function

From a statistical standpoint, the data vector x=( x 1 , x 2 ,..., x n ) x=( x 1 , x 2 ,..., x n ) as the outcome of an experiment is a random sample from an unknown population. The goal of data analysis is to identify the population that is most likely to have generated the sample. In statistics, each population is identified by a corresponding probability distribution. Associated with each probability distribution is a unique value of the model’s parameter. As the parameter changes in value, different probability distributions are generated. Formally, a model is defined as the family of probability distributions indexed by the model’s parameters.

Let denote the probability distribution function (PDF) by f( x|θ ) f( x|θ ) that specifies the probability of observing data y given the parameter w. The parameter vector θ=( θ 1 , θ 2 ,..., θ k ) θ=( θ 1 , θ 2 ,..., θ k ) is a vector defined on a multi-dimensional parameter space. If individual observations, x i 's x i 's are statistically independent of one another, then according to the theory of probability, the PDF for the data x=( x 1 , x 2 ,..., x n ) x=( x 1 , x 2 ,..., x n ) can be expressed as a multiplication of PDFs for individual observations,

f( x,θ )=f( x 1 ,θ )f( x 2 ,θ )···f( x n ,θ ), f( x,θ )=f( x 1 ,θ )f( x 2 ,θ )···f( x n ,θ ), L( θ )= ∏ i=1 n f( x i |θ ) . L( θ )= ∏ i=1 n f( x i |θ ) .

To illustrate the idea of a PDF, consider the simplest case with one observation and one parameter, that is, n=k=1 n=k=1 . Suppose that the data x represents the number of successes in a sequence of 10 independent binary trials (e.g., coin tossing experiment) and that the probability of a success on any one trial, represented by the parameter, θ θ is 0.2. The PDF in this case is then given by

f( x|θ=0.2 )= 10! x!( 10−x )! ( 0.2 ) x ( 0.8 ) 10−x ,( x=0.1,...,10 ), f( x|θ=0.2 )= 10! x!( 10−x )! ( 0.2 ) x ( 0.8 ) 10−x ,( x=0.1,...,10 ),

which is known as the binomial probability distribution. The shape of this PDF is shown in the top panel of Figure 1. If the parameter value is changed to say w = 0.7, a new PDF is obtained as f( x|θ=0.7 )= 10! x!( 10−x )! ( 0.7 ) x ( 0.3 ) 10−x ,( x=0.1,...,10 ); f( x|θ=0.7 )= 10! x!( 10−x )! ( 0.7 ) x ( 0.3 ) 10−x ,( x=0.1,...,10 ); whose shape is shown in the bottom panel of Figure 1. The following is the general expression of the binomial PDF for arbitrary values of θ θ and n:

f( x|θ )= n! θ!( n−x )! θ x ( 1−θ ) n−x ,0≤θ≤1,x=0.1,...,n; f( x|θ )= n! θ!( n−x )! θ x ( 1−θ ) n−x ,0≤θ≤1,x=0.1,...,n;

which as a function of y specifies the probability of data y for a given value of the parameter θ θ . The collection of all such PDFs generated by varying parameter across its range (0 - 1 in this case) defines a model.

Figure 1: Binomial probability distributions of sample size n = 10 and probability parameter θ θ = 0.2 (top) and θ θ = 0.7 (bottom).
(MLE.png)

Maximum Likelihood Estimation

Once data have been collected and the likelihood function of a model given the data is determined, one is in a position to make statistical inferences about the population, that is, the probability distribution that underlies the data. Given that different parameter values index different probability distributions (Figure 1), we are interested in finding the parameter value that corresponds to the desired PDF.

The principle of maximum likelihood estimation (MLE), originally developed by R. A. Fisher in the 1920s, states that the desired probability distribution be the one that makes the observed data most likely, which is obtained by seeking the value of the parameter vector that maximizes the likelihood function L( θ ) L( θ ) . The resulting parameter, which is sought by searching the multidimensional parameter space, is called the MLE estimate, denoted by

θMLE=( θ 1 MLE,..., θ k MLE ) . θMLE=( θ 1 MLE,..., θ k MLE ) .

Let p equal the probability of success in a sequence of Bernoulli trials or the proportion of the large population with a certain characteristic. The method of moments estimate for p is relative frequency of success (having that characteristic). It will be shown below that the maximum likelihood estimate for p is also the relative frequency of success.

Suppose that X is b( 1,p ) b( 1,p ) so that the p.d.f. of X is f( x;p )= p x ( 1−p ) 1−x ,x=0,1,0≤p≤1. f( x;p )= p x ( 1−p ) 1−x ,x=0,1,0≤p≤1. Sometimes is written p∈Ω=[ p:0≤p≤1 ] , p∈Ω=[ p:0≤p≤1 ] , where Ω Ω is used to represent parameter space, that is, the space of all possible values of the parameter. A random sample X 1 , X 2 ,..., X n X 1 , X 2 ,..., X n is taken, and the problem is to find an estimator u( X 1 , X 2 ,..., X n ) u( X 1 , X 2 ,..., X n ) such that u( x 1 , x 2 ,..., x n ) u( x 1 , x 2 ,..., x n ) is a good point estimate of p, where x 1 , x 2 ,..., x n x 1 , x 2 ,..., x n are the observed values of the random sample. Now the probability that X 1 , X 2 ,..., X n X 1 , X 2 ,..., X n takes the particular values is

P( X 1 = x 1 ,..., X n = x n )= ∏ i=1 n p x i ( 1−p ) 1− x i =p ∑ x i ( 1−p ) n− ∑ x i , P( X 1 = x 1 ,..., X n = x n )= ∏ i=1 n p x i ( 1−p ) 1− x i =p ∑ x i ( 1−p ) n− ∑ x i ,

which is the joint p.d.f. of X 1 , X 2 ,..., X n X 1 , X 2 ,..., X n evaluated at the observed values. One reasonable way to proceed towards finding a good estimate of p is to regard this probability (or joint p.d.f.) as a function of p and find the value of p that maximizes it. That is, find the p value most likely to have produced these sample values. The joint p.d.f., when regarded as a function of p, is frequently called the likelihood function. Thus here the likelihood function is:

L( p )=L( p; x 1 , x 2 ,..., x n )=f( x 1 ;p )f( x 2 ;p )···f( x n ;p )= p ∑ x i ( 1−p ) n− ∑ x i ,0≤p≤1. L( p )=L( p; x 1 , x 2 ,..., x n )=f( x 1 ;p )f( x 2 ;p )···f( x n ;p )= p ∑ x i ( 1−p ) n− ∑ x i ,0≤p≤1.

To find the value of p that maximizes L( p ) L( p ) first take its derivative for 0<p<1: 0<p<1:

dL( p ) dp =( ∑ x i ) p n− ∑ x i ( 1−p ) n− ∑ x i −( n− ∑ x i ) p ∑ x i ( 1−p ) n− ∑ x i −1 . dL( p ) dp =( ∑ x i ) p n− ∑ x i ( 1−p ) n− ∑ x i −( n− ∑ x i ) p ∑ x i ( 1−p ) n− ∑ x i −1 .

Setting this first derivative equal to zero gives p ∑ x i ( 1−p ) n− ∑ x i [ ∑ x i p − n− ∑ x i 1−p ]=0. p ∑ x i ( 1−p ) n− ∑ x i [ ∑ x i p − n− ∑ x i 1−p ]=0.

Since 0<p<1 0<p<1 , this equals zero when ∑ x i p − n− ∑ x i 1−p =0. ∑ x i p − n− ∑ x i 1−p =0. Or, equivalently, p= ∑ x i n = x ¯ . p= ∑ x i n = x ¯ .

The corresponding statistics, namely ∑ X i /n= X ¯ ∑ X i /n= X ¯ , is called the maximum likelihood estimator and is denoted by p ^ p ^ ,that is, p ^ = 1 n ∑ i=1 n X i = X ¯ . p ^ = 1 n ∑ i=1 n X i = X ¯ .

When finding a maximum likelihood estimator, it is often easier to find the value of parameter that minimizes the natural logarithm of the likelihood function rather than the value of the parameter that minimizes the likelihood function itself. Because the natural logarithm function is an increasing function, the solution will be the same. To see this, the example which was considered above gives for 0<p<1 0<p<1 ,

ln⁡L( p )=( ∑ i=1 n x i )ln⁡p+( n− ∑ i=1 n x i )ln⁡( 1−p ). ln⁡L( p )=( ∑ i=1 n x i )ln⁡p+( n− ∑ i=1 n x i )ln⁡( 1−p ).

To find the maximum, set the first derivative equal to zero to obtain

d[ ln⁡L( p ) ] dp =( ∑ i=1 n x i )( 1 p )+( n− ∑ i=1 n x i )( −1 1−p )=0, d[ ln⁡L( p ) ] dp =( ∑ i=1 n x i )( 1 p )+( n− ∑ i=1 n x i )( −1 1−p )=0,

which is the same as previous equation. Thus the solution is p= x ¯ p= x ¯ and the maximum likelihood estimator for p is p ^ = X ¯ . p ^ = X ¯ .

Motivated by the preceding illustration, the formal definition of maximum likelihood estimators is presented. This definition is used in both the discrete and continuous cases. In many practical cases, these estimators (and estimates) are unique. For many applications there is just one unknown parameter. In this case the likelihood function is given by L( θ )= ∏ i=1 n f( x i ,θ ) . L( θ )= ∏ i=1 n f( x i ,θ ) .

SEE ALSO:

Maximum Likelihood Estimation - Examples

Content actions

Give Feedback:

E-mail the module author | Rate module ( How does the rating system work?)

Rating system

Ratings

Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

How to rate a module

Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

(0 ratings)

Download:

Module PDF

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections directly in Connexions. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need a Connexions account to use 'My Favorites'.

| A lens (?)

Definition of a lens

Lenses

A lens is a custom view of Connexions content. You can think of it as a fancy kind of list that will let you see Connexions through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to Connexions materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual Connexions member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks