Differential Entropy
Summary: In this module we consider differential entropy.
Consider the entropy of continuous random variables. Whereas the (normal) entropy is the entropy of a discrete random variable, the differential entropy is the entropy of a continuous random variable.
Differential Entropy
Definition 1:
Differential entropy
Usually the logarithm is taken to be base 2, so that the unit of the differential entropy is bits/symbol. Note that is the discrete case,
The differential entropy h X h X X X f x f x
h X = - ∫ - ∞ ∞ f x log f x d x
h
X
x
f x
f x
Now, consider a calculating the differential entropy of some random variables.
Example 1
Consider a uniformly distributed random variable X X c c
c + Δ
c
Δ
1 Δ
1
Δ
c c
c + Δ
c
Δ
We can then find its differential entropy as follows,
h X = - ∫ c c + Δ 1 Δ log 1 Δ d x = log Δ
h
X
x c c Δ
1 Δ
1 Δ
Δ Δ Δ Δ Δ
Example 2
Consider a normal distributed random variable X X m m σ 2 σ 2
1 2 π σ 2 e - x - m 2 2 σ 2
1
2
σ
2
e
x m 2
2
σ 2
We can then find its differential entropy as follows, first calculate
- log f x f x
- log f x = 1 2 log 2 π σ 2 + log e x - m 2 2 σ 2
f x
1
2
2
σ
2
e
x m 2
2
σ 2
E X - m 2 = σ 2
E
X
m
2
σ
2
h X = 1 2 log 2 π σ 2 + 1 2 log e = 1 2 log 2 π e σ 2
h
X
1
2
2
σ
2
1
2
e
1
2
2
e
σ
2
Properties of the differential entropy
In the section we list some properties of the differential entropy.
- The differential entropy can be negative
-
h X + c = h X -
h a X = h X + log | a |
Comments, questions, feedback, criticisms?
Discussion forum
Send feedback
More about this content: Metadata | Version History | Cite This Content
This work is licensed by Anders Gjendemsjø. See the Creative Commons License about permission to reuse this material.
Last edited by Anders Gjendemsjø on Aug 1, 2006 11:34 am GMT-5.