As mentioned earlier, how much a source can be compressed should be related to its entropy. In 1948, Claude E. Shannon introduced three theorems and developed very rigorous mathematics for digital communications. In one of the three theorems, Shannon relates entropy to the minimum number of bits per second required to represent a source without much loss (or distortion).

Consider a source that is modeled by a discrete-time and discrete-valued random process X 1 X 1 , X 2 X 2 , …, X n X n , … where x i ∈ a 1 a 2 … a N x i a 1 a 2 … a N and define p X i ⁢ x i = a j = p j p X i x i a j p j for j= 1 , 2 , … , N j 1 , 2 , … , N , where it is assumed that X 1 X 1 , X 2 X 2 ,… X n X n are mutually independent and identically distributed.

Consider a sequence of length nn

The symbol a 1 a 1 can occur with probability p 1 p 1 . Therefore, in a sequence of length nn, on the average, a 1 a 1 will appear n⁢ p 1 n p 1 times with high probabilities if nn is very large.

Therefore,

where p i =P⁢ X j = a i p i P X j a i for all jj and for all ii.

A typical sequence X X may look like

where a i a i appears n⁢ p i n p i times with large probability. This is referred to as a typical sequence. The probability of X X being a typical sequence is where H⁢X H X is the entropy of the random variables X 1 X 1 , X 2 X 2 ,…, X n X n .

For large nn, almost all the output sequences of length nn of the source are equally probably with probability≃2−(n⁢H⁢X) probability 2 n H X . These are typical sequences. The probability of nontypical sequences are negligible. There are Nn N n different sequences of length nn with alphabet of size NN. The probability of typical sequences is almost 1.

The essence of source coding or data compression is that as n→∞ n , nontypical sequences never appear as the output of the source. Therefore, one only needs to be able to represent typical sequences as binary codes and ignore nontypical sequences. Since there are only 2n⁢H⁢X 2 n H X typical sequences of length nn, it takes n⁢H⁢X n H X bits to represent them on the average. On the average it takes H⁢X H X bits per source output to represent a simple source that produces independent and identically distributed outputs.

The source coding theorem proves existence of source coding techniques that achieve rates close to the entropy but does not provide any algorithms or ways to construct such codes.

If the source is not i.i.d. (independent and identically distributed), but it is stationary with memory, then a similar theorem applies with the entropy H⁢X H X replaced with the entropy rate H=limit   n → ∞ H⁢ X n | X 1 X 2 … X n-1 H n H X n | X 1 X 2 … X n-1

In the case of a source with memory, the more the source produces outputs the more one knows about the source and the more one can compress.

Although Shannon's results are not constructive, there are a number of source coding algorithms for discrete time discrete valued sources that come close to Shannon's bound. One such algorithm is the Huffman source coding algorithm. Another is the Lempel and Ziv algorithm.

Huffman codes and Lempel and Ziv apply to compression problems where the source produces discrete time and discrete valued outputs. For cases where the source is analog there are powerful compression algorithms that specify all the steps from sampling, quantizations, and binary representation. These are referred to as waveform coders. JPEG, MPEG, vocoders are a few examples for image, video, and voice, respectively.

Content actions

Footer

This work is licensed by Behnaam Aazhang under a Creative Commons Attribution License (CC-BY 1.0), and is an Open Educational Resource.

Last edited by Elizabeth Gregory on Oct 3, 2005 1:45 pm GMT-5.