16 Cost Function for Logistic Regression

Negative Log-Likelihood as Cost Function :

$$J(\theta )=-\ell (\theta )=-\sum _i\left[y_i\log \sigma (x_i)+(1-y_i)\log (1-\sigma (x_i))\right]$$

Where:

• $J(\theta )$ : The cost function to be minimised.
• $\sigma (x_i)=\frac{1}{1+\exp (-{\theta }^{\top }{x}_i)}$ : Sigmoid function.
• $y_i$ : Binary target label (0 or 1)

Optimisations :

• Minimise $J(\theta )$ to find the optimal parameters $\hat{\theta }$.
• Use Gradient Descent.

Gradient of Cost Function : $\nabla J(\theta )$ whose $j$-th component is:

$$\frac{\partial J(\theta )}{\partial {\theta }_j}=\sum _{i=1}^n\left[\sigma (x_i)-y_i\right]x_{ij}$$