Automatic differentiation begins with a simple object: a function.
| Section | Title |
|---|---|
| 1 | Chapter 2. Mathematical Foundations |
| 2 | Multivariate Calculus |
| 3 | Jacobians and Hessians |
| 4 | Chain Rule as Composition Algebra |
| 5 | Computational Graphs |
| 6 | Linearization |
| 7 | Dual Spaces and Pushforwards |
| 8 | Taylor Expansions |
Chapter 2. Mathematical FoundationsAutomatic differentiation begins with a simple object: a function.
Multivariate CalculusAutomatic differentiation is usually applied to functions with many inputs and many outputs. The calculus needed for this setting is multivariate calculus: the study of how a...
Jacobians and HessiansThe gradient is enough when a function has many inputs and one scalar output. More general programs need more general derivative objects. Two of the most important are the...
Chain Rule as Composition AlgebraThe chain rule is the central theorem behind automatic differentiation. Every useful AD algorithm is a disciplined way of applying the chain rule to a program.
Computational GraphsA computational graph represents a calculation as nodes and edges. Nodes represent operations or values. Edges represent data dependencies. Automatic differentiation uses this...
LinearizationLinearization is the operation of replacing a nonlinear function by its best local linear approximation at a chosen point. Automatic differentiation can be understood as a...
Dual Spaces and PushforwardsForward mode and reverse mode propagate different kinds of objects.
Taylor ExpansionsDifferentiation describes how a function changes locally. A Taylor expansion extends this idea by approximating a function with a polynomial around a point.