22 Entropy

Entropy quantifies the uncertainty in a random variable $X$. Given a discrete random variable $X$ with range $R_X=\{x_1,x_2,\dots x_m\}$ and probability mass function $P_X(x)$, the entropy of $X$ is defined as:

$$\begin{array}{rl}H(X)& \equiv \mathbb{E}[I_X(X)]\\ & \equiv -\sum _{i=1}^{m}P(X=x_i){\log }_{b}P(X=x_i)\\ & \equiv \mathbb{E}\left[{\log }_b\left(\frac{1}{P_X(X)}\right)\right]\\ & \equiv -\mathbb{E}[{\log }_b{P}_X(X)]\end{array}$$

Values of $b$

1. $b=2$ → Entropy is in bits

   • Most common in information theory
   • Measures uncertainty in terms of binary decisions
   • Standard when talking about data compression, communication, etc.

2. $b=e$ → Entropy is in nats

   • Used more in mathematics and physics (e.g., statistical mechanics)
   • The natural logarithm simplifies calculus-based derivations

3. $b=10$ → Entropy is in hartleys

   • Rarely used, but historically relevant