10 Bayesian Inference and Probabilistic Linear Regression

Paradigms of Machine Learning

Machine Learning seeks to determine the “best fit” for data, such as deciding the optimal degree of a polynomial regression. There are two primary schools of thought:

Frequentist View

• Parameters $(\theta )$: Considered fixed, unknown constants.
• Methodology: Uses estimators like MLE, confidence is gained through repeated experiments (e.g., cross-validation)

Bayesian View

• Parameters $(\theta )$: Treated as random variables.
• Methodology: Uncertainty in $\theta$ is expressed as a probability distribution based on a single observed dataset.
• Bayes’ Law: Used to update the beliefs as more data arrives:

$$p(\theta |X)=\frac{p(X|\theta )p(\theta )}{p(X)}$$

Essentially: Posterior $\propto$ Likelihood $\times$ Prior.

The Gaussian Distribution

The Gaussian (Normal) distribution is ubiquitous due to the Central Limit Theorem, which states that the sum of many independent variables tends towards a Gaussian shape.

Univariate Gaussian

Defined by mean $(\mu )$ and variance $({\sigma }^2)$:

$$\mathcal{N}(x|\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2}$$

Multivariate Gaussian

For an $N$-dimensional vector $\mathbf{x}$, the distribution uses a mean vector $\mathbf{\mu }$ and an $N\times N$ covariance matrix $\mathbf{\Sigma }$:

$$\mathcal{N}(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })=\frac{1}{(2\pi {)}^{N/2}|\mathbf{\Sigma }{|}^{1/2}}\exp \left\{-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })\right\}$$

Tip

Mahalanobis Distance: The term ${\Delta }^2=(\mathbf{x}-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu })$ represents the distance of a point from the mean, accounting for correlations in the data. It simplifies to Euclidean distance $\mathbf{\Sigma }$ is the identity matrix.

Maximum Likelihood Estimation (MLE)

MLE aims to find parameters ($\mathbf{\mu },\mathbf{\Sigma }$) that make the observed dataset $\mathbf{X}$ most probable.

The Process

1. Likelihood Function: Assuming data points are independent and identically distributed (i.i.d), the likelihood is the product of individual probabilities: $p(\mathbf{X}|\mathbf{\mu },\mathbf{\Sigma })={\prod }_{i=1}^{N}p({\mathbf{x}}_i|\mathbf{\mu },\mathbf{\Sigma })$
2. Log-Likelihood $\mathcal{L}$): We take the natural log because it transforms products into sums, making the differentiation much easier
3. Optimisation: Set the partial derivatives of $\mathcal{L}$ with respect to $\mathbf{\mu }$ and $\mathbf{\Sigma }$ to zero

Results for Gaussian MLE:

• ${\mathbf{\mu }}_{ML}=\frac{1}{N}{\sum }_{i=1}^N{\mathbf{x}}_i$ (The sample mean).
• ${\mathbf{\Sigma }}_{ML}=\frac{1}{N}{\sum }_{i=1}^N({\mathbf{x}}_i-\mathbf{\mu })({\mathbf{x}}_i-\mathbf{\mu }{)}^T$ (The sample covariance).

Linear Regression: OLS vs Probabilistic View

Ordinary Least Squares (OLS)

OLS solves for weights $\mathbf{\omega }$ by minimising sum of squared residuals.

• Design Matrix ($\mathbf{\Phi }$): A matrix where each row represents the basis functions of a data point.
• Normal Equation: ${\mathbf{\omega }}_{OLS}=({\mathbf{\Phi }}^T\mathbf{\Phi }{)}^{-1}{\mathbf{\Phi }}^T\mathbf{y}$.
• Moore-Penrose Pseudo-inverse: The term $({\mathbf{\Phi }}^T\mathbf{\Phi }{)}^{-1}{\mathbf{\Phi }}^T$ is denoted as ${\mathbf{\Phi }}^{†}$.

The Probabilistic View

We assume the target $y$ is a linear combination of inputs plus some unexplained Gaussian noise.

$$y=\hat{y}(\mathbf{x},\mathbf{\omega })+ϵ$$

Where $ϵ\sim \mathcal{N}(0,{\beta }^{-1})$ and $\beta$ is the precision (inverse variance).
This leads to a conditional target distribution

$$p(y|\mathbf{x},\mathbf{\omega },\beta )=\mathcal{N}(y|{\mathbf{\omega }}^T\varphi (\mathbf{x}),{\beta }^{-1})$$

The Equivalence: Why MLE = OLS

Under the assumption of additive Gaussian nose, maximising the likelihood of the regression model is mathematically identical to minimising the sum of squared error in OLS.

Step-by-Step Intuition

1. Write the log-likelihood $\mathcal{L}$ for the regression model.
2. Observe that the core term in $\mathcal{L}$ dependent on weights $\mathbf{\omega }$ is the sum of squared residuals: $R(\mathbf{\omega })=\sum (y_i-{\hat{y}}_i{)}^2$.
3. Because there is a negative sign in front of this term in the log-likelihood, maximising $\mathcal{L}$ requires minimising $R(\mathbf{\omega })$.
4. Therefore, ${\mathbf{\omega }}_{MLE}={\mathbf{\omega }}_{OLS}$.

IMPORTANT

OLS requires no specific distribution assumptions to function as an optimisation tool, but it only gains an MLE interpretation if we assume the noise follows a Gaussian distribution.