35 Bias, Variance and Noise

Regression problem setup

• Let $S$ be a training dataset sampled from some distribution.
• Let $x\in \mathcal{X}$ denote an input and $y\in \mathbb{R}$ the corresponding output.
• Assume the true relationship is given by

$$y=f(x)+ϵ$$

where $ϵ$ is a random noise with $\mathbb{E}[ϵ]=0$ and variance $\text{Var}(\mathbb{E})={\sigma }^2$.

• A learning algorithm produces an estimator $\hat{f}(x;S)$.

The quantity we want to analyse

We want to analyse the performance of a model at a fixed test point $\mathbf{x}$. The total expected squared error at $\mathbf{x}$ is :

$${\mathbb{E}}_{S,ϵ}\left[{\left(y-\hat{f}(\mathbf{x};S)\right)}^2\right]$$

This expectation is taken with respect to :

• The unknown true distribution of the training set $S$ : Randomly drawn from the true distribution $p(\mathbf{x},y)$.
• The noise $ϵ$ : Variability in $y$ for a fixed $\mathbf{x}$.

Bias

The bias is defined as the difference between the true function $f(x)$ and the expected model prediction over all possible training set :

$${\text{Bias}}^2={\left(f(x)-{\mathbb{E}}_S\left[\hat{f}(x;S)\right]\right)}^2$$

where :

• $f(x)$ : The true function that generates the data.
• $\hat{f}(x;S)$ : The model’s prediction at $x$ given training set $S$.
• ${\mathbb{E}}_S\left[\hat{f}(x;S)\right]$ : The expected prediction over different training sets.

Variance

The variance measures how much the model’s predictions vary for different training set $S$.

$$\text{Variance}={\mathbb{E}}_S\left[{\left(\hat{f}(x;S)-{\mathbb{E}}_S\left[\hat{f}(x;S)\right]\right)}^2\right]$$

This captures the sensitivity of the model to different training sets.

Noise

The irreducible noise, ${\sigma }^2$, is the variance in $y$ that is independent of $x$.

$$\text{Irreducible Noise}={\mathbb{E}}_{ϵ}\left[{ϵ}^2\right]={\sigma }^2$$

This is the inherent randomness in the data, which cannot be reduced by any model.

Decomposing the Error while Ignoring the Noise

Start with :

$${\mathbb{E}}_S\left[{\left(\hat{f}(x;S)-f(x)\right)}^2\right]$$

Add/Subtract the mean :

$${\mathbb{E}}_S\left[{\left(\hat{f}(x;S)-{\mathbb{E}}_S[\hat{f}(x;S)]+{\mathbb{E}}_S[\hat{f}(x;S)]-f(x)\right)}^2\right]$$

Expand the square :

$$=\underset{{\text{Bias}}^2}{\underbrace{{\left({\mathbb{E}}_S[\hat{f}(x;S)]-f(x)\right)}^2}}+\underset{\text{Variance}}{\underbrace{{\mathbb{E}}_S\left[{\left(\hat{f}(x;S)-{\mathbb{E}}_S[\hat{f}(x;S)]\right)}^2\right]}}+\text{Cross-term}$$

Eliminate the Cross-term :

$${\mathbb{E}}_S\left[\hat{f}(x;S)-{\mathbb{E}}_S[\hat{f}(x;S)]\right]=0.$$

Including the Noise Term

• Recall that the observed output is : $y=f(x)+ϵ$ has variance ${\sigma }^2$.
• Therefore, the overall expected squared error is :

$$\begin{array}{rl}{\mathbb{E}}_{S,\epsilon }\left[{\left(y-\hat{f}(x;S)\right)}^2\right]& ={\mathbb{E}}_S\left[{\left(f(x)-\hat{f}(x;S)\right)}^2\right]+\underset{{\sigma }^2}{\underbrace{{\mathbb{E}}_{\epsilon }\left[{\epsilon }^2\right]}}\\ & =\underset{{\text{Bias}}^2}{\underbrace{{\left({\mathbb{E}}_S[\hat{f}(x;S)]-f(x)\right)}^2}}+\underset{\text{Variance}}{\underbrace{{\mathbb{E}}_S\left[{\left(\hat{f}(x;S)-{\mathbb{E}}_S[\hat{f}(x;S)]\right)}^2\right]}}+{\sigma }^2\\ & ={\text{Bias}}^2+\text{Variance}+{\sigma }^2.\end{array}$$

• This gives us the full decomposition

$$\boxed{\text{Total Error}={\text{Bias}}^2+\text{Variance}+\text{Noise}.}$$