23 Joint Entropy and Conditional Entropy

Let $X,Y$ be two discrete random variables: Simplified Notation that we will use :

• Marginal PMF of $(X)$: $P_X(\cdot )=P(\cdot )$
• Marginal probability of $“(X=x)”$ : $P(X=x)=P(x)$
• Joint PMF: $P_{X,Y}(\cdot ,\cdot )=P(\cdot ,\cdot )$
• Joint PMF at $(x,y)$: $P(X=x,Y=y)=P(x,y)$
• Conditional PMF: $P_{X|Y}(\cdot \mid \cdot )=P(\cdot \mid \cdot )$
• Conditional PMF at $(x,y)$: $P(X=x\mid Y=y)=P(x\mid y)$ Note: This is a deterministic value – but if we write $P(x\mid Y)$, this is a random variable since $Y$ is a random variable.

Joint Entropy

A measure of the uncertainty associated with a set of variables.. For discrete random variables $X$ and $Y$ :

$$\begin{array}{rl}H(X,Y)& =-\mathbb{E}[\log P(X,Y)]\\ & =-\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\log P(x,y)\end{array}$$

Conditional Entropy

Quantifies uncertainty of the outcome of a random variable $Y$ given the outcome of another random variable $X$.

$$\begin{array}{rl}H(Y\mid X)& =-\mathbb{E}[\log P(Y\mid X)]\\ & =-\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\log P(y\mid x)\end{array}$$

Chain Rule for conditional Entropy

By definition of Conditional Entropy, and the Bayes rule identity for conditional probability, $P(y|x)=\frac{P(x,y)}{P(x)}$, we have :

$$\begin{array}{rl}H(Y\mid X)& =-\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\log P(y\mid x)\\ & =-\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\log \left(\frac{P(x,y)}{P(x)}\right)\\ & =-\sum _{x\in {\mathcal{R}}_X}\sum _{y\in {\mathcal{R}}_Y}P(x,y)\left[\log P(x,y)-\log P(x)\right]\\ & =H(X,Y)-H(X)\end{array}$$

Similarly,

$$H(X\mid Y)=H(X,Y)-H(Y)$$