Entropy Form

The last quantity on the right, conditional entropy H_Y|X, as H_X,Y-H_X. Substituting that difference in place of H_Y|X in :
(1)
In this form, mutual information is the sum of the two self-entropies minus the joint entropy.

We used the mutually associated case to derive Equation 1. However, Equation 1 also applies to the mutually unassociated case. For that case, mutual information turns out to be zero. To see that, we use the equation for the unassociated case. Using that definition, we substitute H_X+H_Y in place of H_X,Y in Equation 1. That gives I_Y;X=H_Y+H_X- (H_X+H_Y), or I_Y;X=0. So, the mutual information of two independent systems is zero. In other words, one system tells us nothing about the other. Incidentally, rearranging Equation 1 provides an alternate expression for joint entropy, H_X,Y:
H_X,Y = H_Y + H_X-I_Y;X....... (2)
Joint entropy for two systems or dimensions, whether mutually related or not, therefore is the sum of the two self-entropies minus the mutual information. (For two unrelated systems, mutual information I_Y;X is zero. Eq. 2 then reduces, H_X,Y=H_Y+H_X.)