First derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is...
| Section | Title |
|---|---|
| 1 | Chapter 8. Higher-Order Differentiation |
| 2 | Hessian Computation |
| 3 | Hessian-Vector Products |
| 4 | Higher-Order Reverse Mode |
| 5 | Nested AD |
| 6 | Taylor Mode AD |
| 7 | Efficient Higher-Order Methods |
| 8 | Perturbation Confusion |
| 9 | Complexity of Higher Orders |
Chapter 8. Higher-Order DifferentiationFirst derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is...
Hessian ComputationFor a scalar function
Hessian-Vector ProductsA Hessian-vector product computes
Higher-Order Reverse ModeReverse mode is efficient for scalar-output functions because it propagates one adjoint backward through the computation and produces a full gradient. For
Nested ADNested automatic differentiation means applying automatic differentiation inside another automatic differentiation computation.
Taylor Mode ADTaylor mode automatic differentiation computes derivatives by propagating truncated Taylor series through a program.
Efficient Higher-Order MethodsHigher-order derivatives contain rich geometric information, but naïve computation quickly becomes impractical.
Perturbation ConfusionPerturbation confusion is a correctness bug that appears in nested automatic differentiation, especially nested forward mode. It happens when two derivative computations...
Complexity of Higher OrdersHigher-order automatic differentiation faces a fundamental problem: derivative structure grows combinatorially with order.