Skip to content Skip to navigation Skip to collection information

OpenStax_CNX

You are here: Home » Content » Introduction to Statistics » Maximum Likelihood Estimation (MLE)

Navigation

Lenses

What is a lens?

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

This content is ...

In these lenses

  • Statistics

    This collection is included inLens: Mathieu Plourde's Lens
    By: Mathieu Plourde

    Click the "Statistics" link to see all content selected in this lens.

Recently Viewed

This feature requires Javascript to be enabled.
 

Maximum Likelihood Estimation (MLE)

Module by: Ewa Paszek. E-mail the author

Summary: This course is a short series of lectures on Introductory Statistics. Topics covered are listed in the Table of Contents. The notes were prepared by Ewa Paszek and Marek Kimmel. The development of this course has been supported by NSF 0203396 grant.

MAXIMUM LIKELIHOOD ESTIMATION (MLE)

Likelihood function

From a statistical standpoint, the data vector x=( x 1 , x 2 ,..., x n ) x=( x 1 , x 2 ,..., x n ) as the outcome of an experiment is a random sample from an unknown population. The goal of data analysis is to identify the population that is most likely to have generated the sample. In statistics, each population is identified by a corresponding probability distribution. Associated with each probability distribution is a unique value of the model’s parameter. As the parameter changes in value, different probability distributions are generated. Formally, a model is defined as the family of probability distributions indexed by the model’s parameters.

Let denote the probability distribution function (PDF) by f( x|θ ) f( x|θ ) that specifies the probability of observing data y given the parameter w. The parameter vector θ=( θ 1 , θ 2 ,..., θ k ) θ=( θ 1 , θ 2 ,..., θ k ) is a vector defined on a multi-dimensional parameter space. If individual observations, x i 's x i 's are statistically independent of one another, then according to the theory of probability, the PDF for the data x=( x 1 , x 2 ,..., x n ) x=( x 1 , x 2 ,..., x n ) can be expressed as a multiplication of PDFs for individual observations,

f( x,θ )=f( x 1 ,θ )f( x 2 ,θ )···f( x n ,θ ), f( x,θ )=f( x 1 ,θ )f( x 2 ,θ )···f( x n ,θ ), L( θ )= i=1 n f( x i |θ ) . L( θ )= i=1 n f( x i |θ ) .

To illustrate the idea of a PDF, consider the simplest case with one observation and one parameter, that is, n=k=1 n=k=1 . Suppose that the data x represents the number of successes in a sequence of 10 independent binary trials (e.g., coin tossing experiment) and that the probability of a success on any one trial, represented by the parameter, θ θ is 0.2. The PDF in this case is then given by

f( x|θ=0.2 )= 10! x!( 10x )! ( 0.2 ) x ( 0.8 ) 10x ,( x=0.1,...,10 ), f( x|θ=0.2 )= 10! x!( 10x )! ( 0.2 ) x ( 0.8 ) 10x ,( x=0.1,...,10 ),

which is known as the binomial probability distribution. The shape of this PDF is shown in the top panel of Figure 1. If the parameter value is changed to say w = 0.7, a new PDF is obtained as f( x|θ=0.7 )= 10! x!( 10x )! ( 0.7 ) x ( 0.3 ) 10x ,( x=0.1,...,10 ); f( x|θ=0.7 )= 10! x!( 10x )! ( 0.7 ) x ( 0.3 ) 10x ,( x=0.1,...,10 ); whose shape is shown in the bottom panel of Figure 1. The following is the general expression of the binomial PDF for arbitrary values of θ θ and n:

f( x|θ )= n! θ!( nx )! θ x ( 1θ ) nx ,0θ1,x=0.1,...,n; f( x|θ )= n! θ!( nx )! θ x ( 1θ ) nx ,0θ1,x=0.1,...,n;

which as a function of y specifies the probability of data y for a given value of the parameter θ θ . The collection of all such PDFs generated by varying parameter across its range (0 - 1 in this case) defines a model.

Figure 1: Binomial probability distributions of sample size n = 10 and probability parameter θ θ = 0.2 (top) and θ θ = 0.7 (bottom).
 (MLE.png)

Maximum Likelihood Estimation

Once data have been collected and the likelihood function of a model given the data is determined, one is in a position to make statistical inferences about the population, that is, the probability distribution that underlies the data. Given that different parameter values index different probability distributions (Figure 1), we are interested in finding the parameter value that corresponds to the desired PDF.

The principle of maximum likelihood estimation (MLE), originally developed by R. A. Fisher in the 1920s, states that the desired probability distribution be the one that makes the observed data most likely, which is obtained by seeking the value of the parameter vector that maximizes the likelihood function L( θ ) L( θ ) . The resulting parameter, which is sought by searching the multidimensional parameter space, is called the MLE estimate, denoted by

θMLE=( θ 1 MLE,..., θ k MLE ) . θMLE=( θ 1 MLE,..., θ k MLE ) .

Let p equal the probability of success in a sequence of Bernoulli trials or the proportion of the large population with a certain characteristic. The method of moments estimate for p is relative frequency of success (having that characteristic). It will be shown below that the maximum likelihood estimate for p is also the relative frequency of success.

Suppose that X is b( 1,p ) b( 1,p ) so that the p.d.f. of X is f( x;p )= p x ( 1p ) 1x ,x=0,1,0p1. f( x;p )= p x ( 1p ) 1x ,x=0,1,0p1. Sometimes is written pΩ=[ p:0p1 ] , pΩ=[ p:0p1 ] , where Ω Ω is used to represent parameter space, that is, the space of all possible values of the parameter. A random sample X 1 , X 2 ,..., X n X 1 , X 2 ,..., X n is taken, and the problem is to find an estimator u( X 1 , X 2 ,..., X n ) u( X 1 , X 2 ,..., X n ) such that u( x 1 , x 2 ,..., x n ) u( x 1 , x 2 ,..., x n ) is a good point estimate of p, where x 1 , x 2 ,..., x n x 1 , x 2 ,..., x n are the observed values of the random sample. Now the probability that X 1 , X 2 ,..., X n X 1 , X 2 ,..., X n takes the particular values is

P( X 1 = x 1 ,..., X n = x n )= i=1 n p x i ( 1p ) 1 x i =p x i ( 1p ) n x i , P( X 1 = x 1 ,..., X n = x n )= i=1 n p x i ( 1p ) 1 x i =p x i ( 1p ) n x i ,

which is the joint p.d.f. of X 1 , X 2 ,..., X n X 1 , X 2 ,..., X n evaluated at the observed values. One reasonable way to proceed towards finding a good estimate of p is to regard this probability (or joint p.d.f.) as a function of p and find the value of p that maximizes it. That is, find the p value most likely to have produced these sample values. The joint p.d.f., when regarded as a function of p, is frequently called the likelihood function. Thus here the likelihood function is:

L( p )=L( p; x 1 , x 2 ,..., x n )=f( x 1 ;p )f( x 2 ;p )···f( x n ;p )= p x i ( 1p ) n x i ,0p1. L( p )=L( p; x 1 , x 2 ,..., x n )=f( x 1 ;p )f( x 2 ;p )···f( x n ;p )= p x i ( 1p ) n x i ,0p1.

To find the value of p that maximizes L( p ) L( p ) first take its derivative for 0<p<1: 0<p<1:

dL( p ) dp =( x i ) p n x i ( 1p ) n x i ( n x i ) p x i ( 1p ) n x i 1 . dL( p ) dp =( x i ) p n x i ( 1p ) n x i ( n x i ) p x i ( 1p ) n x i 1 .

Setting this first derivative equal to zero gives p x i ( 1p ) n x i [ x i p n x i 1p ]=0. p x i ( 1p ) n x i [ x i p n x i 1p ]=0.

Since 0<p<1 0<p<1 , this equals zero when x i p n x i 1p =0. x i p n x i 1p =0. Or, equivalently, p= x i n = x ¯ . p= x i n = x ¯ .

The corresponding statistics, namely X i /n= X ¯ X i /n= X ¯ , is called the maximum likelihood estimator and is denoted by p ^ p ^ ,that is, p ^ = 1 n i=1 n X i = X ¯ . p ^ = 1 n i=1 n X i = X ¯ .

When finding a maximum likelihood estimator, it is often easier to find the value of parameter that minimizes the natural logarithm of the likelihood function rather than the value of the parameter that minimizes the likelihood function itself. Because the natural logarithm function is an increasing function, the solution will be the same. To see this, the example which was considered above gives for 0<p<1 0<p<1 ,

lnL( p )=( i=1 n x i )lnp+( n i=1 n x i )ln( 1p ). lnL( p )=( i=1 n x i )lnp+( n i=1 n x i )ln( 1p ).

To find the maximum, set the first derivative equal to zero to obtain

d[ lnL( p ) ] dp =( i=1 n x i )( 1 p )+( n i=1 n x i )( 1 1p )=0, d[ lnL( p ) ] dp =( i=1 n x i )( 1 p )+( n i=1 n x i )( 1 1p )=0,

which is the same as previous equation. Thus the solution is p= x ¯ p= x ¯ and the maximum likelihood estimator for p is p ^ = X ¯ . p ^ = X ¯ .

Motivated by the preceding illustration, the formal definition of maximum likelihood estimators is presented. This definition is used in both the discrete and continuous cases. In many practical cases, these estimators (and estimates) are unique. For many applications there is just one unknown parameter. In this case the likelihood function is given by L( θ )= i=1 n f( x i ,θ ) . L( θ )= i=1 n f( x i ,θ ) .

Collection Navigation

Content actions

Download:

Collection as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Module as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Add:

Collection to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks

Module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks