03 Nonlinear Transformation in Machine Learning

The Need for Nonlinear Transformations

Standard linear models, like basic Logistic Regression, assume that a straight line (a linear decision boundary where $w^{T}x=0$) can separate classes.
In many real-world scenarios, classes are nonlinearly separable. Forcing a straight line on such data leads to underfitting, where the model is too simple to capture the underlying structure. By transforming the data, we can use the power of linear algorithms to solve nonlinear problems.

Feature Mapping & Basis Functions

To solve nonlinear problems, we apply a transformation function $\varphi (\mathbf{x})$ to the input vector $\mathbf{x}$. This creates a new feature space.

Polynomial Basis Expansion

A common transformation is to create polynomials of degree $p$. This expansion includes all possible terms up to that degree based on the original variables.

• Example: Degree 2 Expansion (1 Input Variable)
  • Original: $\mathbf{x}=(1,x_1{)}^T$
  • Transformed: $\varphi (\mathbf{x})=(1,x_1,x_1^2{)}^T$
• Example: Degree 2 Expansion (2 Input Variables)
  • Original: $\mathbf{x}=(1,x_1,x_2{)}^T$
  • Transformed: $\varphi (\mathbf{x})=(1,x_1,x_2,x_1^2,x_2^2,x_1{x}_2{)}^T$

Tip

Key Insight: A linear decision boundary in the high-dimensional feature space $(w^T\varphi (\mathbf{x})=0)$ translates back to a nonlinear decision boundary (like a parabola or circle) in the original input spac.

Non-Polynomial Transformations

Transformations are not limited to polynomials. You can use any nonlinear function that helps separate the data, such as exponential functions:

$$x=(1,x_1)\to \varphi (x)=(1,x_1,e^{x_1}{)}^T$$

Implementation in Logistic Regression

When using transformation, the core logic of Logistic Regression remains the same, but every instance of $\mathbf{x}$ is replaced by $\varphi (\mathbf{x})$.

Mathematical Adaptation:

• Transformed Logit: $\text{logit}(p_1)=w^T\varphi (\mathbf{x})$
• Probability Model: $p(1|\varphi (\mathbf{x}),w)=\frac{e^{w^T\varphi (\mathbf{x})}}{1+e^{w^T\varphi (\mathbf{x})}}$

Optimisation Formulas

To find the optimal weights $\mathbf{w}$, we minimise the error function $E(\mathbf{w})$ using the transformed training set $\mathcal{T}=\{(\varphi ({\mathbf{x}}^{(i)}),y^{(i)}){\}}_{i=1}^N$

• Gradient: $\nabla E(\mathbf{w})={\sum }_{i=1}^N(p(1|\varphi ({\mathbf{x}}^{(i)}),\mathbf{w})-y^{(i)})\varphi ({\mathbf{x}}^{(i)})$
• Hessian: $HE(\mathbf{w})={\sum }_{i=1}^{N}p(1|\varphi ({\mathbf{x}}^{(i)}),\mathbf{w})(1-p(1|\varphi ({\mathbf{x}}^{(i)}),\mathbf{w}))\varphi ({\mathbf{x}}^{(i)})\varphi ({\mathbf{x}}^{(i)}{)}^T$

The Linearity Paradox

Is a model with nonlinear transformation still a “linear model” ?
Yes. In machine learning the “linearity” of a model typically refers to its parameters $\mathbf{w}$ and not the input variables $\mathbf{x}$.

• The model is nonlinear in the input space (it draws curves).
• The model is linear in the feature space (it draws a straight line in the embedding).
• The model is linear in its parameters (weights are not squared or used in exponents).

Advantages and Risks

Advantages

• Efficiency: We can continue using well-understood and fast linear learning algorithms.
• Robustness: Linear models often have strong generalisation properties.

Caveats & Common Mistakes

• Overfitting: As the polynomial degree $p$ increases, the model gains more “power” and might start fitting to noise in the training data rather than the actual trend.
• Dimensionality Explosion: The number of features in $\varphi (\mathbf{x})$ can grow very quickly as you increase the number of input variables and the degree of expansion, making calculations slower.

Step-by-Step Adoption Process

1. Define $\varphi (\mathbf{x})$: Choose the basis functions (e.g., quadratic expansion).
2. Data Transformation: Apply $\varphi (\mathbf{x})$ to all training samples ${\mathbf{x}}^{(i)}$ to create a new dataset of $(\varphi (\mathbf{x}),y)$ pairs.
3. Train Linear Model: Use standard algorithms (like Gradient Descent) to find weights $\mathbf{w}$ for the transformed data.
4. Predict: To classify a new point, first transform it to $\varphi (\mathbf{x})$, then apply the linear model ${\mathbf{w}}^T\varphi (\mathbf{x})$.