Differential Entropy

Differential entropy differs from normal or absolute entropy in that the random variable need not be discrete. Given a continuous random variable  with a probability density function , the differential entropy  is defined as

(1)

When we have a continuous random vector  that consists of  random variables , , ..., , the differential entropy of  is defined as the -fold integral

			(2)
			(3)

where  is the joint probability density function of .

Thus, for example, the differential entropy of a multivariate Gaussian random variate  with covariance matrix  is

			(4)
			(5)

Additional properties of differential entropy include

(6)

where  is a constant and

(7)

where  is a scaling factor and  is a scalar random variable. The above property can be generalized to the case of a random vector  premultiplied by a matrix ,

(8)

where  is the determinant of matrix .

REFERENCES:

Cover, T. M. and Thomas, J. A. Elements of Information Theory. New York: Wiley, 1991.