26 Mutual Information

Mutual Information measures the information that two random variables $X$ and $Y$ share.

Intuition : It quantifies how much knowing one variable reduces uncertainty about the other. For two discrete random variables $X$ and $Y$, the mutual, information is defined as :

$$I(X;Y)=\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\log \left(\frac{P(x,y)}{P(x)P(y)}\right)$$

Mutual Information as KL Divergence

The mutual information $I(X;Y)$ can be expressed as the Kullback-Leibler (KL) divergence between the joint distribution and the product of marginal distributions :

$$\begin{array}{rl}I(X;Y)& =\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\log \left(\frac{P(x,y)}{P(x)P(y)}\right)\\ I(X;Y)& =D_{KL}\left(P(X,Y)\Vert P_X{P}_Y\right)\end{array}$$

Interpretation : Mutual information measures the divergence (‘error’) of modelling the joint probability distribution $P_{(X,Y)}$ as the product of marginals $P_X{P}_Y$.

Special Case : When $X$ and $Y$ are independent.

$$P(x,y)=P(x)P(y)\Rightarrow I(X;Y)=D_{KL}(P(X,Y)\Vert P_X{P}_Y)=0$$

Relationship/identities

$$\begin{array}{rl}I(X;Y)& \equiv H(X)-H(X|Y)\\ & \equiv H(Y)-H(Y|X)\\ & \equiv H(X)+H(Y)-H(X,Y)\\ & \equiv H(X,Y)-H(Y|X)-H(X|Y)\end{array}$$

These equations reveal how mutual information quantifies the reduction in uncertainty about one random variable given knowledge of another.

Properties of Mutual Information

Key Properties :

• Non-negative : $I(X;Y)\ge 0$
• Symmetric : $I(X;Y)=I(Y;X)$
• Measures statistical dependence :
  • $I(X;Y)=0$ if and only if $X$ and $Y$ are independent.
  • $I(X;Y)$ increases with the dependence between $X$ and $Y$ and with their individual entropies $H(X)$ and $H(Y)$.
• $I(X;X)=H(X)-H(X|X)=H(X)+0=H(X)$ In practice, we use Mutual Information (MI) to decide which features of the data to use for classification. The features that have highest MI with the class label are the informative featues.