14 Logistic Regression

• Also known as logit regression.
• A model that outputs a real number in $[0,1]$, thresholded at $0.5$ to obtain binary outcomes (class 0 or class 1).
• Goal : Predict the probability that an example belongs to the “1” class.
• Application : Widely used in medical and social sciences.
• Approach :
  • Use the so-called log-odds (logit) to create a $linearmodel$.
  • Apply the sigmoid function (logistic function) to convert log-odds to a real number in $[0,1]$ - i.e., a probability.

Logit = Log-Odds

Given an example having $d$ features $\mathbf{x}=(x_1,x_2,\dots ,x_n)$, its target $Y$, is a Bernoulli random variable. Takes value 1 with some probability $p$ and 0 with probability $1-p$.

• Probability : A number $p\in [0,1]$ that describes the likelihood of an event occurring.
• Odds : The ratio of the probability of an event occurring to the probability of it not occurring.

$$\text{odds}=\frac{p}{1-p}$$

• Logit (log-odds) : The logarithm of the odds.

$$\text{logit}(p)=\log \left(\frac{p}{1-p}\right)=\log (p)-\log (1-p)$$

The logit

For the $i$ -th example, $(x_{i,1},x_{i,2},\dots x_{i,d})$,

$$\begin{array}{rl}\text{logit}(p_i)& =\log \left(\frac{p_i}{1-p_i}\right)\\ & ={\theta }_0+{\theta }_1{x}_{i,1}+{\theta }_2{x}_{i,2}+\cdots +{\theta }_d{x}_{i,d}\\ & ={\theta }_0+\sum _{j=1}^d{\theta }_j{x}_{i,j}\\ & ={\theta }^{\top }{x}_i\end{array}$$

where :

• ${\theta }_0$ : Intercept term.
• ${\theta }_1,{\theta }_2,\dots ,{\theta }_d$ : Coefficients (aka. parameters of the model).
• ${\mathbf{x}}_i=(1,x_{i,1},x_{i,2},\dots ,x_{i,d})$; $\theta =({\theta }_1,{\theta }_2,\dots ,{\theta }_d)$
• $p_i=P(Y_i=1\mid x_i)$, where $Y_i$ is the target variable (also called “dependent variable”).

The Sigmoid

Solve:

$$\log \left(\frac{p_i}{1-p_i}\right)={\theta }^{\top }{x}_i$$

We exponentiate both sides:

$$\left(\frac{p_i}{1-p_i}\right)=\exp ({\theta }^{\top }{x}_i)$$

Multiply both sides by $(1-p_i)$:

$$p_i=(1-p_i)\exp ({\theta }^{\top }{x}_i)=\exp ({\theta }^{\top }{x}_i)-p_i\exp ({\theta }^{\top }{x}_i)$$

Bring all $p_i$ terms to one side:

$$(1+\exp ({\theta }^{\top }{x}_i))p_i=\exp ({\theta }^{\top }{x}_i)$$

Solve for $p_i$:

$$p_i=\frac{\exp ({\theta }^{\top }{x}_i)}{1+\exp ({\theta }^{\top }{x}_i)}=\frac{1}{1+\exp (-{\theta }^{\top }{x}_i)}=:\sigma (x_i)$$

The RHS is the sigmoid function $\sigma (x_i)=\frac{1}{1+\exp (-{\theta }^{\top }{x}_i)}$. It maps the logit (a real value) to a probability (in $[0,1]$). It signifies $p_i=P(Y_i=1|x_i)$, the probability that label 1 is predicted for $x_i$.

Note : It follows that $P(Y_i=0|x_i)=1-\sigma (x_i)$.