21 Self Information

Definition : Given a random variable $X$ with probability mass function $P_X(x)$, the self-information of measuring $X$ as outcome $x$ is defined as :

$$I_X(x)=-{\log }_b\left[P_X(x)\right]={\log }_b\left(\frac{1}{P_X(x)}\right)$$

Self-information quantifies the level of surprise in observing a particular outcome $x$ for a random variable $X$ that has PMF $P_X(x)$.

Units : The base $b$ of the logarithm determines the units of self-information :

• $b=2$ : Measured in bits.
• $b=e$ : Measured in natural units or nats.
• $b=10$ : Measured in dits, bans or hartleys.

Self-Information and Log-Odds

• Self-information and logit (log-odds) are related concepts.
• For an event $A$, occurring with probability $p$ hence the probability of the event not occurring $P(\neg A)=1-P(A)$, the logit function is defined as :

$$\text{logit}(p)=\log \left(\frac{p}{1-p}\right)=\log (p)-\log (1-p)$$

Relationship :

$$\text{logit}(A)=I(\neg A)-I(A)$$

where $I(A)$ and $I(\neg A)$ represent the self-information of events $A$ and $\neg A$ respectively.