01 Supervised Learning

Supervised learning involves finding a final hypothesis $g$ that approximates an unknown target function $f$.

Core Data Structures

• Input Space ($\mathcal{X}$): A $d$ dimensional space containing features. Inputs can be Numeric (age), Ordinal (low/medium/high), or Categorical (car brands).
• Output Space ($\mathcal{Y}$): The target values (e.g., house prices for regression or categories for classification).
• Training Set ($\mathcal{T}$): A collection of $N$ input-output pairs: $\mathcal{T}=\{(x^{(1)},y^{(1)}),\dots ,(x^{(N)},y^{(N)})\}$.

The Design Matrix

To process data efficiently, all input vectors from the training set are often organised into a Design Matrix.

• Each row represents one training example $x^{(i)}$.
• Each column represents a specific independent variable (feature).
• A “bias” column of 1s is often added as the first column $(x_0=1)$ to account for the intercept term $w_0$.

$$\text{Design matrix}\mathbf{X}=\left(\begin{array}{cccc}{x}_1^{(1)}& x_2^{(1)}& \cdots & x_d^{(1)}\\ x_1^{(2)}& x_2^{(2)}& \cdots & x_d^{(2)}\\ ⋮& ⋮& \ddots & ⋮\\ x_1^{(N)}& x_2^{(N)}& \cdots & x_d^{(N)}\end{array}\right)$$ $$\text{Vector independent variables}\mathbf{y}=\left(\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ⋮\\ y^{(N)}\end{array}\right)$$

Logistic Regression: Hypothesis Set

Despite its name, Logistic Regression is for classification, specifically binary classification $(\mathcal{Y}=\{0,1\})$.
From Linear Scores to Probabilities

1. The Score: We calculate a linear combination of inputs $z=w^{T}x$.
2. The Problem: $w^{T}x$ is unbounded ($-\infty$ to $\infty$), but the probabilities must be $[0,1]$
3. The Solution (Logit): We model the logit (log-odds) as the linear combination:

$$\text{logit}(p_1)=\ln \left(\frac{p_1}{1-p_1}\right)=w^{T}x$$

4. The Activation (Sigmoid): Solving for $p_1$ gives us the Sigmoid function which Squashes the score into a probability:

$$p_1=\sigma (w^{T}x)=\frac{e^{w^{T}x}}{1+e^{w^{T}x}}=\frac{1}{1+e^{-w^{T}x}}$$

Decision Boundary

• The model predicts Class 1 is $p_1\ge 0.5$ (which occurs when $w^{T}x\ge 0$).
• The model predicts Class 0 if $p_1<0.5$ (which occurs when $w^{T}x<0$).
• Decision Boundary: The hyperplane is defined by $w^{T}x=0$.

Tip

Distance and Confidence: The larger the absolute value of the score $|w^{T}x|$, the further the point is from the decision boundary, and the higher the model’s confidence in its pediction.