02 Bayesian Estimation

Let $D$ denote data, and $\theta$ the parameters of our model.

• Bayesian represent uncertainty by treating $\theta$ as a random variable with prior distribution. Parameters and data are often continuous-values.
• let p($\theta$) be the prior PDF (Probability density function).
• In simple terms. Once the data is observed, we want to update our belief about $\theta$ given the new evidence. This updated belief is represented by the posterior distribution, denoted as $P(\theta \mid data)$
• Return the posterior distribution of $\theta$ given data $D$ :

$$p(\theta |D)=\frac{p(D|\theta )p(\theta )}{p(D)}$$

• Bayesian estimation does not operate through optimisation to obtain best parameter values.
• $p(D)=\int p(D|\theta )p(\theta )d\theta$ is called marginal likelihood, or evidence.
• Predictive distributions for a new example:
  • in supervised learning: $p(Y|X,D)=\int p(Y|X,\theta )p(\theta |D)d\theta$
  • in unsupervised learning: $p(X|D)=\int p(X|\theta )p(\theta |D)d\theta$