Introduction Sufficient statistics arise in nearly every aspect of statistical inference. It is important to understand them before progressing to areas such as hypothesis testing and parameter estimation. Suppose we observe an N-dimensional random vector X, characterized by the density or mass function f θ x , where θ is a p-dimensional vector of parameters to be estimated. The functional form of f x is assumed known. The parameter θ completely determines the distribution of X. Conversely, a measurement x of X provides information about θ through the probability law f θ x . Suppose X X 1 X 2 , where X i θ 1 are IID. Here θ is a scalar parameter specifying the mean. The distribution of X is determined by θ through the density f θ x 1 2 x 1 θ 2 2 1 2 x 2 θ 2 2 On the other hand, if we observe x 100 102 , then we may safely assume θ 0 is highly unlikely. The N-dimensional observation X carries information about the p-dimensional parameter vector θ. If p N , one may ask the following question: Can we compress x into a low-dimensional statistic without any loss of information? Does there exist some function t T x , where the dimension of t is M N , such that t carries all the useful information about θ? If so, for the purpose of studying θ we could discard the raw measurements x and retain only the low-dimensional statistic t. We call t a sufficient statistic. The following definition captures this notion precisely: Let X 1 , … , X M be a random sample, governed by the density or probability mass function f θ x . The statistic T x is sufficient for θ if the conditional distribution of x, given T x t , is independent of θ. Equivalently, the functional form of f θ t x does not involve θ. How should we interpret this definition? Here are some possibilities:

1. Let f θ x t denote the joint density or probability mass function on ( X , T ( X ) ) . If T X is a sufficient statistic for θ, then f θ x f θ x T x f θ t x f θ t f t x f θ t Therefore, the parametrization of the probability law for the measurement x is manifested in the parametrization of the probability law for the statistic T x .

2. Given t T x , full knowledge of the measurement x brings no additional information about θ. Thus, we may discard x and retain on the compressed statistic t.

3. Any inference strategy based on f θ x may be replaced by a strategy based on f θ t .

Binary Information Source (Scharf, pp.78) Suppose a binary information source emits a sequence of binary (0 or 1) valued, independent variables x 1 , … , x N . Each binary symbol may be viewed as a realization of a Bernoulli trial: x n Bernoulli θ , iid. The parameter θ 0 1 is to be estimated. The probability mass function for the random sample x x 1 … x N is f θ x n 1 N f θ x n n 1 N θ f θ x x n 1 θ 1 x n θ k 1 θ N k where k n 1 N x n is the number of 1's in the sample. We will show that k is a sufficient statistic for x. This will entail showing that the conditional probability mass function f θ k x does not depend on θ. The distribution of the number of ones in N independent Bernoulli trials is binomial: f θ k N k θ k 1 θ N k Next, consider the joint distribution of ( x , x n ) . We have f θ x f θ x x n Thus, the conditional probability may be written f θ k x f θ x k f θ k f θ x f θ k θ k 1 θ N k N k θ k 1 θ N k 1 N k This shows that k is indeed a sufficient statistic for θ. The N values x 1 , … , x N can be replaced by the quantity k without losing information about θ.

In the previous example, suppose we wish to store in memory the information we possess about θ. Compare the savings, in terms of bits, we gain by storing the sufficient statistic k instead of the full sample x 1 , … , x N .