11 Bayesian Linear Regression

Frequentist vs Bayesian Paradigms

In linear regression, we seek to find the parameters $\theta$ (or weights $\omega$) that best maps predictors to targets.

Frequentist Approach (MLE & OLS)

• View: Parameters $\theta$ are fixed but unknown
• Estimation: Computed using estimators like Maximum Likelihood Estimation (MLE) or Ordinary Least Squares (OLS).
• Validation: Confidence is evaluated through repeated experiments or cross-validation
• Assumption: MLE typically assumes additive Gaussian noise: $y-X\theta \sim \text{Gaussian}(0,{\sigma }^{2}I)$

Bayesian Approach

• View: Parameters $\theta$ are random variables.
• Estimation: We start with a prior belief and update it using a single observed dataset to form a posterior distribution.
• Benefit: Naturally handles undetermined systems (e.g., trying to fit a line through only one point), where a frequentist model would have infinite solutions.

The Bayesian Framework

The goal is to move from a prior belief about weights to a posterior distribution after seeing data.

The Components

• Prior $p(\omega )$: Our belief about the weights before seeing any data. Often assumed to be Gaussian: $p(\omega )=\mathcal{N}(\omega |0,{\alpha }^{-1}I)$.
• Likelihood $p(y|X,\omega ,\beta )$: The probability of observing the targets $y$ given the parameters and predictors.
• Posterior $p(\omega |y,X)$: The updated belief after observing data.

Info

$\text{Posterior}\propto \text{Likelihood}\times \text{Prior}$

Mathematical Solution for Posterior

If we use a conjugate prior (a Gaussian prior for a Gaussian likelihood), the posterior is also a Gaussian Distribution

$$p(\omega |y,X)=\mathcal{N}(\omega |m_{post},S_{post})$$

The Posterior Parameters

• Mean: $m_{post}=\beta S_{post}{\Phi }^{T}y$
• Covariance: $S_{post}^{-1}=\alpha I+\beta {\Phi }^T\Phi$
• Note: $\Phi$ is the design matrix, $\alpha$ is the prior precision, and $\beta$ is the noise precision.

Predictive Distribution and Uncertainty

Bayesian regression doesn’t just give one “best” line, it gives a family of possible lines sampled from the posterior.

• Increasing Observations $(N)$: As the number of data points increases, the “spread” of possible lines narrows.
• Convergence: With enough data $(N\to \infty )$, the posterior distribution becomes very sharp, and the model uncertainty reduces significantly.

MAP Estimation and Regularisation

Maximum A Posteriori (MAP) estimation involves finding the weight $\omega$ that maximises the posterior distribution

Connection to Ridge Regression

Maximum A Posteriori (MAP) estimation involves finding the weight $\omega$ that maximises the posteriori distribution

Connection to Ridge Regression

When we take the log of the posterior with a Gaussian prior, we get a cost function to minimise

$$L=\frac{1}{2}\sum _{i=1}^N\{y_i-{\omega }^T\varphi (x_i){\}}^2+\frac{\alpha }{2\beta }{\omega }^T\omega$$

• The First Term: Sum of squared residuals (standard OLS objective).
• The Second Term: The Regularisation Term ($\frac{\alpha }{2\beta }{\omega }^T\omega$), also known as a quadratic regulariser
• Intuition: This term “shrinks” the weights toward zero, which prevents the model from over-fitting to noise, especially when data is scarce.

Tip

MAP estimation for Bayesian linear regression with a Gaussian prior is identical to Regularised Least Squares or Ridge Regression.