Differential Entropy
1.3
2004/02/20 12:05:26 US/Central
2006/08/01 11:34:00.645 GMT-5
Anders
Gjendemsjø
gjendems@iet.ntnu.no
Anders
Gjendemsjø
gjendems@iet.ntnu.no
Differential
Entropy
Information
Theory
In this module we consider differential entropy.
Consider the entropy of continuous random variables. Whereas the (normal) entropy is the entropy of a discrete random variable, the differential entropy is the entropy of a continuous random variable.
Differential Entropy
Differential entropy
The differential entropy hX of a continuous random variable
X with a pdf fx is defined as
h
X
x
fx
fx
Usually the logarithm is taken to be base 2, so that the unit of the differential entropy is bits/symbol. Note that is the discrete case,
hX depends only on the pdf of X. Finally, we note that
the differential entropy is the expected value of
f
x
, i.e.,
h
X
E
f
x
Now, consider a calculating the differential entropy of some random variables.
Consider a uniformly distributed random variable X from c to
c
Δ
.
Then its density is
1
Δ
from c to
c
Δ
, and zero otherwise.
We can then find its differential entropy as follows,
h
X
x
c
cΔ
1Δ
1Δ
Δ
Note that by making Δ arbitrarily small, the differential entropy can be made arbitrarily negative,
while taking Δ arbitrarily large, the differential entropy becomes arbitrarily positive.
Consider a normal distributed random variable X, with mean m and
variance σ2.
Then its density is
1
2
σ
2
e
xm
2
2
σ2
.
We can then find its differential entropy as follows, first calculate
fx:
fx
1
2
2
σ
2
e
xm
2
2
σ2
Then since
E
X
m
2
σ
2
,
we have
h
X
1
2
2
σ
2
1
2
e
1
2
2
e
σ
2
Properties of the differential entropy
In the section we list some properties of the differential entropy.
- The differential entropy can be negative
-
h
X
c
h
X
, that is translation does not change the differential entropy.
-
h
a
X
h
X
a
, that is scaling does change the differential entropy.
The first property is seen from both and . The two latter can be shown by using
.