34 Estimator

Suppose we have many training sets of size $n$ generated from the some unknown distribution. From each we estimate a separate $\widehat{{\theta }_n}$.

Bias of an Estimator :

• Definition : Bias of $\widehat{{\theta }_n}$ is $\text{Bias}(\widehat{{\theta }_n})=\mathbb{E}[{\hat{\theta }}_n]-{\theta }^{\ast }$
• An estimator is unbiased if $\mathbb{E}[{\hat{\theta }}_n]={\theta }^{\ast }$ for all ${\theta }^{\ast }$.

Variance of an Estimator :

• Definition : Variance of $\widehat{{\theta }_n}$ is $\text{Var}({\hat{\theta }}_n)=\mathbb{E}\left[({\hat{\theta }}_n-{\theta }^{\ast }{)}^2\right]$ (if $\hat{\theta }$ is a scalar variable) or $\mathbb{C}\text{ov}[{\hat{\theta }}_n]$ (if $\hat{\theta }$ is a vector variable).
• Unlike bias, the variance does not directly depend on the true parameter ${\theta }^{\ast }$.

Bias-Variance decomposition of the Mean Squared Error

We look at the expected square error (expectation over the distribution that generated the training sets) from the true parameter value ${\theta }^{\ast }$.

$$\mathbb{E}\left[(\hat{\theta }-{\theta }^{\ast }{)}^2\right]=\mathbb{E}[{\hat{\theta }}^2]-2\mathbb{E}[\hat{\theta }]{\theta }^{\ast }+({\theta }^{\ast }{)}^2$$

Add & Subtract $\mathbb{E}[\hat{\theta }{]}^2$ to complete the square :

$$=\mathbb{E}[{\hat{\theta }}^2]+({\theta }^{\ast }-\mathbb{E}[\hat{\theta }]{)}^2-(\mathbb{E}[\hat{\theta }]{)}^2$$

Rearrange to conclude :

$$\mathbb{E}[(\hat{\theta }-{\theta }^{\ast }{)}^2]=\underset{\text{Bias}[\hat{\theta }{]}^2}{\underbrace{({\theta }^{\ast }-\mathbb{E}[\hat{\theta }]{)}^2}}+\underset{\text{Var}[\hat{\theta }]}{\underbrace{\mathbb{E}[(\hat{\theta }-\mathbb{E}[\hat{\theta }]{)}^2]}}$$

MLE and MAP estimators

The Maximum Likelihood Estimator for $\theta$ is :

$${\theta }_{\text{MLE}}=\frac{{\sum }_{i=1}^n{X}_i}{n}$$

The MLE is a sample mean of Bernoulli trials : The Maximum a Posteriori (MAP) Estimator for $\theta$ is :

$${\theta }_{\text{MAP}}=\frac{{\sum }_{i=1}^n{X}_i+\alpha -1}{n+\alpha +\beta -2}$$

The MAP is the maximiser of the posterior distribution of $\theta$ when the prior distribution is $p(\theta )=Beta(\alpha ,\beta )$. Note : Bias-variance is a frequentist concept. While MAP(and Bayesian mean) derive from Bayesian framework, all estimators can be analysed in the frequentist framework.

Bias of the Estimator :

Bias is defined as :

$$\text{Bias}(\hat{\theta })=\mathbb{E}[\hat{\theta }]-\theta$$

Bias of MLE :

$$\text{Bias}({\hat{\theta }}_{\text{MLE}})=\mathbb{E}[{\hat{\theta }}_{\text{MLE}}]-\theta =\mathbb{E}\left[\frac{{\sum }_{i=1}^n{X}_i}{n}\right]-\theta \text{\hspace{1em}(1)}$$ $$=\frac{\mathbb{E}\left[{\sum }_{i=1}^n{X}_i\right]}{n}-\theta =\frac{n\theta }{n}-\theta =\theta -\theta =0\text{\hspace{1em}(2)}$$

Thus, the MLE is unbiased. Bias of MAP :

$$\text{Bias}({\hat{\theta }}_{\text{MAP}})=\mathbb{E}[{\hat{\theta }}_{\text{MAP}}]-\theta =\frac{n\theta +\alpha -1}{n+\alpha +\beta -2}-\theta$$

Thus, the MAP estimator is biased towards the prior mean, especially for small $n$, but becomes unbiased as $n$ grows large.

Variance of the Estimators :

Variance is given by:

$$\text{Var}(\hat{\theta })=\mathbb{E}[{\hat{\theta }}^2]-{\left(\mathbb{E}[\hat{\theta }]\right)}^2$$

Variance of MLE :

$$\text{Var}({\hat{\theta }}_{\text{MLE}})=\text{Var}\left(\frac{{\sum }_{i=1}^n{X}_i}{n}\right)=\frac{\text{Var}\left({\sum }_{i=1}^n{X}_i\right)}{n^2}=\frac{n\theta (1-\theta )}{n^2}=\frac{\theta (1-\theta )}{n}$$

Variance if MAP Estimator :

$$\text{Var}({\hat{\theta }}_{\text{MAP}})=\frac{1}{(n+\alpha +\beta -2{)}^2}\cdot \text{Var}\left(\sum _{i=1}^n{X}_i\right)=\frac{n\theta (1-\theta )}{(n+\alpha +\beta -2{)}^2}$$

This can be made smaller than the $\text{Var}({\hat{\theta }}_{\text{MLE}})$ by an informative prior ($\alpha ,\beta$ away from 1). Notice the trade-off with $Bias({\hat{\theta }}_{\text{MAP}})$. For large $n$, this becomes approx. $\text{Var}({\hat{\theta }}_{\text{MAP}})\approx \frac{\theta (1-\theta )}{n+\alpha +\beta -2}$. Similar to the MLE, but with a denominator that includes prior information ($\alpha \text{and}\beta$).

Implications of Bias-Variance Analysis

• We want estimators that have low bias and low variance - but both are not achievable simultaneously with a finite sample, and there are trade-offs.
• Bias-variance properties of estimators can guide the choice of estimator to use. MLE has low bias if $n$ is sufficiently large, but it has high variance.
• High bias, low variance estimators (e.g., regularization methods - also interpretable as MAP estimators) improve stability and generalization.
• Bayesian estimators incorporate prior information, introducing bias but reducing variance. The Bayesian posterior mean is often biased but achieves lower MSE than frequentist estimators.
• The best choice depends on sample size, prior knowledge, and application needs.