Forward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value...
| Section | Title |
|---|---|
| 1 | Chapter 5. Forward Mode Automatic Differentiation |
| 2 | Dual Numbers |
| 3 | Forward Evaluation Rules |
| 4 | Jacobian-Vector Products |
| 5 | Complexity Analysis |
| 6 | Higher-Dimensional Tangent Spaces |
| 7 | Efficient Seeding Strategies |
| 8 | Sparse Forward Methods |
| 9 | Case Studies |
Chapter 5. Forward Mode Automatic DifferentiationForward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value...
Dual NumbersDual numbers give forward mode automatic differentiation a compact algebraic form. Instead of storing a value and a tangent as two unrelated fields, we package them into one...
Forward Evaluation RulesForward mode automatic differentiation works by replacing each primitive operation with an extended operation on pairs:
Jacobian-Vector ProductsThe natural output of forward mode automatic differentiation is a Jacobian-vector product. Instead of constructing the full Jacobian matrix explicitly, forward mode computes...
Complexity AnalysisForward mode automatic differentiation has a simple cost model. It evaluates the original program and, at the same time, evaluates the tangent program. Each primitive...
Higher-Dimensional Tangent SpacesSo far, forward mode has propagated a single tangent direction:
Efficient Seeding StrategiesForward mode automatic differentiation computes Jacobian-vector products:
Sparse Forward MethodsMany real-world Jacobians are sparse. Most derivative entries are zero because outputs depend only on small subsets of inputs.
Case StudiesForward mode automatic differentiation appears in many numerical systems where directional derivatives, local sensitivities, or small parameter sets are important. This...