08 Multi-Layer Perceptron (MLP) Architecture

The MLP, also known as a Feed-Forward Net, is described as a “soft perceptron” that utilises gradient descent to learn it parameters.

Mathematical Notation

• $L$: Total number of layers. Layer $1$ is the input layer, and the superscript $L$ denotes the output layer.
• $m$: The width(number of units) of a layer. This can vary between layers.
• $w_{jk}^l$: The weight of the connection from the $k$-th unit in layer $l-1$ to the $j$-th unit in layer $l$.
• $b_j^l$: The bias of the $j$-th unit in layer $l$
• $z_j^l$: The weighted input to unit $j$ in layer $l$: $z_j^l={\sum }_k{w}_{jk}^l{a}_k^{l-1}+b_j^l$
• $a_j^l$: The activation of unit $j$ in layer $l$: $a_l^j=\sigma (z_j^l)$, where $\sigma$ is an activation function.

Parameter Calculation

For a layer of width $m$ connected to a previous layer of the same width, the total number of parameters (weights + bias) is calculated as $m^2+m$.

Activation Functions

The goal is to replace the standard non-differentiable sign function with a different non-linear function to enable gradient-based learning.

• Sigmoid Function: $\sigma (x)=\frac{1}{1+\exp (-x)}$
• Mapping: It maps values from $(-\infty ,+\infty )$ to the range $(0,1)$
• Derivative: ${\sigma }^{\prime }(x)=\sigma (x)(1-\sigma (x))$