13 Gradient Descent

Gradient Descent is a first-order iterative optimization algorithm for finding the local minimum of a differentiable cost function. Key Idea : Employ the negative gradient at each step to decrees the cost function.

Two ingredients :

• Direction determined by the gradient at the point.
• Magnitude also called the step size or learning rate. Intuition :
• Strat at any value of parameter $\theta$.
• Change $\theta$ in the direction that decreases the cost function.
• Repeat each step until the decrease in cost with each step is very small.

Formally, we define the negative gradient of a cost function $J(\theta )$ as:

$$-{\nabla }_{\theta }J=-\frac{dJ(\theta )}{d\theta }.$$

We choose a step size parameter $\eta$ (learning rate).
The update equation becomes:

$$\begin{array}{rl}\theta (t+1)& =\theta (t)-\eta {\nabla }_{\theta }J(\theta (t))\text{(1)}\\ & =\theta (t)-\eta \frac{dJ(\theta (t))}{d\theta }\text{(2)}\end{array}$$

Example

If $J(\theta )=-\log ({\theta }^5(1-\theta {)}^7)$, what’s the gradient ?

Ans :

$$J(\theta )=-\left(5\log (\theta )+7\log (1-\theta )\right)$$

Taking the derivative with respect to $\theta$:

$$\frac{dJ}{d\theta }=-\left(\frac{5}{\theta }+7\cdot \frac{1}{1-\theta }\cdot (-1)\right)=-\left(\frac{5}{\theta }-\frac{7}{1-\theta }\right)$$

Final simplified gradient:

$${\nabla }_{\theta }J(\theta )=\frac{7}{1-\theta }-\frac{5}{\theta }$$