Automatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full...
| Section | Title |
|---|---|
| 1 | Chapter 4. Core Theory of Automatic Differentiation |
| 2 | Elementary Operations |
| 3 | Forward Accumulation |
| 4 | Reverse Accumulation |
| 5 | Mixed-Mode Differentiation |
| 6 | Computational Complexity |
| 7 | Numerical Exactness up to Floating Point |
| 8 | AD as Program Transformation |
| 9 | Summary and Synthesis |
| 10 | Tangent Propagation |
Chapter 4. Core Theory of Automatic DifferentiationAutomatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full...
Elementary OperationsAutomatic differentiation reduces differentiation to a finite collection of elementary operations. Every program, regardless of complexity, is decomposed into primitive...
Forward AccumulationForward accumulation is the forward-mode form of automatic differentiation. It propagates derivative information in the same order as ordinary program evaluation. Each...
Reverse AccumulationReverse accumulation is the reverse-mode form of automatic differentiation. It propagates derivative information backward from outputs to inputs.
Mixed-Mode DifferentiationMixed-mode differentiation combines forward accumulation and reverse accumulation in the same derivative computation. It is used when neither pure forward mode nor pure...
Computational ComplexityAutomatic differentiation is fundamentally a computational technique. Its practical importance comes from the fact that derivatives can often be computed with asymptotic cost...
Numerical Exactness up to Floating PointAutomatic differentiation computes derivatives exactly with respect to the executed floating point program. This distinguishes AD from numerical differentiation, which...
AD as Program TransformationAutomatic differentiation can be understood as a transformation from one program into another program.
Summary and SynthesisAutomatic differentiation is a method for computing derivatives by transforming programs into derivative-propagating computations. It does not approximate derivatives...
Tangent PropagationForward mode automatic differentiation computes derivatives by propagating tangent values alongside ordinary values. The ordinary value is called the primal. The derivative...