Consider the entropy of
continuous random variables. Whereas the (normal)
entropy is the entropy of a
discrete random variable, the differential entropy is the entropy of a continuous random variable.
Differential Entropy
Definition 1:
Differential entropy
The differential entropy
hXhX of a continuous random variable
XX with a pdf
fxfx is defined as
hX=-∫-∞∞fxlogfxdx
h
X
x
fx
fx
(1)
Usually the logarithm is taken to be base 2, so that the unit of the differential entropy is bits/symbol. Note that is the discrete case,
hXhX depends only on the pdf of
XX. Finally, we note that
the differential entropy is the expected value of
-logfx
f
x
, i.e.,
hX=-Elogfx
h
X
E
f
x
(2)
Now, consider a calculating the differential entropy of some random variables.
Example 2 Consider a normal distributed random variable XX, with mean mm and
variance σ2σ2.
Then its density is
12πσ2e-x-m22σ2
1
2
σ
2
e
xm
2
2
σ2
.
We can then find its differential entropy as follows, first calculate
-logfxfx:
-logfx=12log2πσ2+logex-m22σ2
fx
1
2
2
σ
2
e
xm
2
2
σ2
(4)
Then since
EX-m2=σ2
E
X
m
2
σ
2
,
we have
hX=12log2πσ2+12loge=12log2πeσ2
h
X
1
2
2
σ
2
1
2
e
1
2
2
e
σ
2
(5)
Properties of the differential entropy
In the section we list some properties of the differential entropy.
- The differential entropy can be negative
-
hX+c=hX
h
X
c
h
X
, that is translation does not change the differential entropy.
-
haX=hX+log|a|
h
a
X
h
X
a
, that is scaling does change the differential entropy.
The first property is seen from both
Example 1 and
Example 2. The two latter can be shown by using
Equation 1.