Reverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a...
| Section | Title |
|---|---|
| 1 | Chapter 6. Reverse Mode Automatic Differentiation |
| 2 | Reverse Computational Graphs |
| 3 | Vector-Jacobian Products |
| 4 | Reverse Accumulation Algorithms |
| 5 | Tape-Based Systems |
| 6 | Wengert Lists |
| 7 | Memory-Time Tradeoffs |
| 8 | Checkpointing |
| 9 | Reverse Mode in Deep Learning |
| 10 | Case Studies |
Chapter 6. Reverse Mode Automatic DifferentiationReverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a...
Reverse Computational GraphsReverse mode automatic differentiation operates on a computational graph. The forward pass evaluates the graph from inputs to outputs. The reverse pass traverses the same...
Vector-Jacobian ProductsReverse mode automatic differentiation fundamentally computes vector-Jacobian products. The gradient of a scalar function is a special case of this more general operation.
Reverse Accumulation AlgorithmsReverse accumulation is the operational core of reverse mode automatic differentiation. The forward pass evaluates a program and records dependency information. The reverse...
Tape-Based SystemsMost reverse mode automatic differentiation systems require a mechanism for recording the forward computation so that the reverse pass can later traverse it backward. This...
Wengert ListsA Wengert list is a linear representation of a computation in which every intermediate result is assigned to a unique variable. It is one of the earliest and most influential...
Memory-Time TradeoffsReverse mode automatic differentiation is computationally efficient for scalar-output functions, but it has a major systems cost: it needs information from the forward pass...
CheckpointingCheckpointing is a technique for reducing the memory cost of reverse mode automatic differentiation by selectively storing intermediate states and recomputing missing values...
Reverse Mode in Deep LearningReverse mode automatic differentiation is the mathematical and systems basis of backpropagation. In deep learning, the objective is usually a scalar loss depending on many...
Case StudiesThis section studies reverse mode automatic differentiation through concrete examples. Each case has the same structure: