18 Training Optimisation and Normalisation

Learning Rate Scheduling

Start with a large learning rate and reduce it during training (e.g., every $n$ epochs or when validation loss plateaus).

Importance of Normalisation

Network performs best when data distributions are consistent.

Input Normalisation Techniques

• Feed Scaling: Dividing pixel values $(0-255)$ by $255$ to get a $0-1$ range.
• Min-Max Norm: Scaling data specifically between its maximum and minimum values.
• Z-Score Norm: $(x-\mu )/\sigma$. This centres at $0$ with standard deviation of $1$.

Batch Normalisation (BN):

• Concept: Normalising the output of every neuron or filter within a batch.
• Learnable Parameters: BN introduces $\gamma$(scaling) and $\beta$(shifting) which the network learns to fund the optimal distribution.
• Test Time Difficulty: During testing, there is no “batch”.
  • Solution: Precompute the mean $(\mu )$ and standard deviation $(\sigma )$ during training and use those fixed values during testing
• Efficacy: BN significantly speeds up training and improves validation accuracy.