Skip to content Skip to navigation

Connexions

You are here: Home » Content » The Fisher-Neyman Factorization Theorem

Navigation

Recently Viewed

This feature requires Javascript to be enabled.
 

The Fisher-Neyman Factorization Theorem

Module by: Clayton Scott, Robert Nowak. E-mail the authors

Determining a sufficient statistic directly from the definition can be a tedious process. The following result can simplify this process by allowing one to spot a sufficient statistic directly from the functional form of the density or mass function.

Theorem 1: Fisher-Neyman Factorization Theorem

Let f θ x f θ x be the density or mass function for the random vector x x, parametrized by the vector θ θ. The statistic t=T⁢x t T x is sufficient for θ θ if and only if there exist functions a⁢x a x (not depending on θ θ) and b θ t b θ t such that f θ x=a⁢x⁢b θ t f θ x a x b θ t for all possible values of x x.

In an earlier example we computed a sufficient statistic for a binary communication source (independent Bernoulli trials) from the definition. Using the above result, this task becomes substantially easier.

Example 1

Bernoulli Trials Revisited

Suppose x n ∼Bernoulli⁢θ x n Bernoulli θ are IID, ∀ n ,n= 1 , … , N : n= 1 , … , N n n 1 , … , N . Denote x= x 1 … x n T x x 1 … x n . Then

f θ x=∏ n =1Nθ x n ⁢1−θ1− x n =θk⁢1−θN−k=a⁢x⁢b θ k f θ x n 1 N θ x n 1 θ 1 x n θ k 1 θ N k a x b θ k
(1)
where k=∑ n =1N x n k n 1 N x n , a⁢x=1 a x 1 , and b θ k=θk⁢1−θN−k b θ k θ k 1 θ N k . By the Fisher-Neyman factorization theorem, k k is sufficient for θ θ.

The next example illustrates the appliction of the theorem to a continuous random variable.

Example 2

Normal Data with Unknown Mean

Consider a normally distributed random sample x 1 , … , x N ∼𝒩θ1 x 1 , … , x N θ 1 , IID, where θθ is unknown. The joint pdf of x= x 1 … x n T x x 1 … x n is f θ x=∏ n =1Nf θ x n =12⁢πN2⁢e-12⁢∑ n =1N x n −θ2 f θ x n 1 N f θ x n 1 2 N 2 -1 2 n 1 N x n θ 2 We would like to rewrite f θ x f θ x is the form of a⁢x⁢b θ t a x b θ t , where dim⁢t<N dim t N . At this point we require a trick-one that is commonly used when manipulating normal densities, and worth remembering. Define x-=1N⁢∑ n =1N x n x 1 N n 1 N x n , the sample mean. Then

f θ x=12⁢πN2⁢e-12⁢∑ n =1N x n −x-+x-−θ2=12⁢πN2⁢e-12⁢∑ n =1N x n −x-2+2⁢( x n −x-)⁢(x-−θ)+x-−θ2 f θ x 1 2 N 2 -1 2 n 1 N x n x x θ 2 1 2 N 2 -1 2 n 1 N x n x 2 2 x n x x θ x θ 2
(2)
Now observe
∑ n =1N( x n −x-)⁢(x-−θ)=(x-−θ)⁢∑ n =1N x n −x-=(x-−θ)⁢(x-−x-)=0 n 1 N x n x x θ x θ n 1 N x n x x θ x x 0
(3)
so the middle term vanishes. We are left with f θ x=12⁢πN2⁢e-12⁢∑ n =1N x n −x-2⁢e-12⁢∑ n =1Nx-−θ2 f θ x 1 2 N 2 -1 2 n 1 N x n x 2 -1 2 n 1 N x θ 2 where a⁢x=12⁢πN2⁢e-12⁢∑ n =1N x n −x-2 a x 1 2 N 2 -1 2 n 1 N x n x 2 , b θ t=e-12⁢∑ n =1Nx-−θ2 b θ t -1 2 n 1 N x θ 2 , and t=x t x . Thus, the sample mean is a one-dimensional sufficient statistic for the mean.

Proof of Theorem

First, suppose t=T⁢x t T x is sufficient for θ θ. By definition, f θ | T⁢x=t x f θ T x t x is independent of θ θ. Let f θ xt f θ x t denote the joint density or mass function for ( X , T ( X ) ) ( X , T ( X ) ) . Observe f θ x=f θ xt f θ x f θ x t . Then

f θ x=f θ xt=f θ | t x⁢f θ t=a⁢x⁢b θ t f θ x f θ x t f θ t x f θ t a x b θ t
(4)
where a⁢x=f θ | t x a x f θ t x and b θ t=f θ t b θ t f θ t . We prove the reverse implication for the discrete case only. The continuous case follows a similar argument, but requires a bit more technical work (Scharf, pp.82; Kay, pp.127).

Suppose the probability mass function for xx can be written f θ x=a⁢x⁢b θ x f θ x a x b θ x where t=T⁢x t T x . The probability mass function for tt is obtained by summing f θ xt f θ x t over all xx such that T⁢x=t T x t :

f θ t=∑T⁢x=tf θ xt=∑T⁢x=tf θ x=∑T⁢x=ta⁢x⁢b θ t f θ t x T x t f θ x t x T x t f θ x x T x t a x b θ t
(5)
Therefore, the conditional mass function of xx, given tt, is
f θ | t x=f θ xtf θ t=f θ xf θ t=a⁢x∑T⁢x=ta⁢x f θ t x f θ x t f θ t f θ x f θ t a x x T x t a x
(6)
This last expression does not depend on θθ, so tt is a sufficient statistic for θθ. This completes the proof.

Remark:

From the proof, the Fisher-Neyman factorization gives us a formula for the conditional probability of xx given tt. In the discrete case we have fx| t =a⁢x∑T⁢x=ta⁢x f t x a x x T x t a x An analogous formula holds for continuous random variables (Scharf, pp.82).

Further Examples

The following exercises provide additional examples where the Fisher-Neyman factorization may be used to identify sufficient statistics.

Exercise 1

Uniform Measurements

Suppose x 1 , … , x N x 1 , … , x N are independent and uniformly distributed on the interval θ 1 θ 2 θ 1 θ 2 . Find a sufficient statistic for θ= θ 1 θ 2 T θ θ 1 θ 2 .

Hint:
Express the likelihood f θ x f θ x in terms of indicator functions.

Exercise 2

Poisson

Suppose x 1 , … , x N x 1 , … , x N are independent measurements of a Poisson random variable with intensity parameter θθ: ∀ x ,x= 0 , 1 , 2 , … :f θ x=e−θ⁢θxx! x x 0 , 1 , 2 , … f θ x θ θ x x

2.a)

Find a sufficient statistic tt for θθ.

2.b)

What is the conditional probability mass function of xx, given tt, where x= x 1 … x N T x x 1 … x N ?

Exercise 3

Normal with Unknown Mean and Variance

Consider x 1 , … , x N ∼𝒩μσ2 x 1 , … , x N μ σ 2 , IID, where θ 1 =μ θ 1 μ and θ 2 =σ2 θ 2 σ 2 are both unknown. Find a sufficient statistic for θ= θ 1 θ 2 T θ θ 1 θ 2 .

Hint:
Use the same trick as in Example 2.

References

  1. L. Scharf. (1991). Statistical Signal Processing. Addison-Wesley.
  2. Kay. (1993). Estimation Theory. Prentice Hall.

Content actions

Download module as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks