9 items
A computational graph is a graph that represents a numerical computation. The nodes represent values or operations. The edges describe how data flows from one operation to the next.
Gradient computation is the process of measuring how a scalar output changes when its input values change. In deep learning, the scalar output is usually the loss, and the inputs are usually the model parameters.
Reverse-mode differentiation is the method used by backpropagation. It computes derivatives by first evaluating a function forward, then propagating gradient information backward from the output to the inputs.
The chain rule is the mathematical rule that makes backpropagation possible. Neural networks are built by composing many functions. The chain rule tells us how to differentiate such compositions.
Gradients are enough for most neural network training. A gradient tells us how a scalar loss changes with respect to parameters.
Backpropagation is the algorithm used to compute gradients in neural networks efficiently.
Gradient flow describes how derivative information moves backward through a neural network during training.
An automatic differentiation engine is the system that records numerical operations and computes derivatives from them.
Deep learning frameworks need a way to represent computation.