09 Computation Graphs

A Computation Graph is directed acyclic graph (DAG) used to represent and compute mathematical functions.

• Nodes Represent variables (inputs/ outputs) or intermediate results which are functions of their predecessors.
• Edges: Indicate the direction of computation

Example

Chain Rule

To handle networks with millions or billions of parameters, calculating derivatives by hand is impossible; the Chain Rule allows for efficient algorithmic computation.

Single Path Chain Rule

If $f$ is a function of $g$, and $g$ is a function of $t$, then: $\frac{\partial f}{\partial t}=\frac{\partial f}{\partial g}\frac{\partial g}{\partial t}$

Multivariate Chain Rule

When an output is influenced by multiple paths (multiple successors), the derivatives must be summed:

• For two paths: $\frac{\partial f}{\partial t}=\frac{\partial f}{\partial g}\frac{\partial g}{\partial t}+\frac{\partial f}{\partial h}\frac{\partial h}{\partial t}$
• General Case: $\frac{\partial f}{\partial t}={\sum }_i\frac{\partial f}{\partial g_i}\frac{\partial g_i}{\partial t}$, where $g_i$ are successors of $t$.