15 Maximum Likelihood Estimation of Logistic Regression

Goal : Estimate the unknown parameters $\theta$ of the logistic regression model.

Likelihood Function:

$$L(\theta \mid y,x)=\prod _{i}P(y_i\mid x_i;\theta )$$

For binary outcomes:

$$P(y_i\mid x_i;\theta )=\sigma (x_i{)}^{y_i}{\left(1-\sigma (x_i)\right)}^{1-y_i}$$

where $\sigma (x_i)=\frac{1}{1+\exp (-{\theta }^{\top }{x}_i)}$ is the sigmoid function.

Log-Likelihood Function:

$$\ell (\theta )=\log L(\theta \mid y,x)=\sum _i\left[y_i\log \sigma (x_i)+(1-y_i)\log (1-\sigma (x_i))\right]$$

Estimation :

Find ${\hat{\theta }}_{MLE}$ that minimises the negative log-likelihood:

$${\theta }_{\text{MLE}}=\arg \underset{\theta }{\min }\left[-\ell (\theta )\right]$$