36 Bias-Variance Trade-off

• Bias : Measures the error due to approximating the true function $f(x)$ with the average prediction ${\mathbb{E}}_S[\hat{f}(x;S)]$. $\Rightarrow$ High bias can occur with the overly simple models (underfitting).
• Variance : Measures the variability of the prediction $\hat{f}(x;S)$ around its mean, due to different training sets. $\Rightarrow$ High variance can occur with overly complex models (overfitting).
• Noise : Represents irreducible error inherent in the data generation process.
• Trade-off : Increasing model complexity typically reduces bias but increases variance, and vice versa.