| Chapter | Title | Description |
|---|---|---|
| 1 | Chapter 1. Introduction | A derivative measures how an output changes when an input changes. That sentence is simple, but it is one of the main ideas behind numerical computing, optimization, machine… |
| 2 | Chapter 2. Mathematical Foundations | Automatic differentiation begins with a simple object: a function. |
| 3 | Chapter 3. Programs as Mathematical Objects | A straight-line program is the simplest model of computation used in automatic differentiation. It is a program with a fixed sequence of assignments, no branches, no loops,… |
| 4 | Chapter 4. Core Theory of Automatic Differentiation | Automatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full… |
| 5 | Chapter 5. Forward Mode Automatic Differentiation | Forward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value… |
| 6 | Chapter 6. Reverse Mode Automatic Differentiation | Reverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a… |
| 7 | Chapter 7. Dual Numbers and Algebraic Structures | Dual numbers give the cleanest algebraic model of forward mode automatic differentiation. They extend ordinary real numbers with a formal infinitesimal part. Instead of… |
| 8 | Chapter 8. Higher-Order Differentiation | First derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is… |
| 9 | Chapter 9. Differentiation of Control Flow | A conditional is a program construct that chooses one computation among several possible computations. In ordinary code, this is written as if, else, switch, case, pattern… |
| 10 | Chapter 10. Matrix and Tensor Differentiation | Matrix calculus is the notation and rule system used to differentiate functions whose inputs, outputs, or intermediate values are vectors, matrices, or tensors. Automatic… |
| 11 | Chapter 11. Compiler and Runtime Design | Source transformation is an implementation strategy for automatic differentiation in which a program that computes a function is rewritten into another program that computes… |
| 12 | Chapter 12. AD in Modern Programming Languages | Lisp is one of the natural homes of automatic differentiation. It treats programs as data, has a simple expression syntax, and supports macro systems that can transform code… |
| 13 | Chapter 13. Optimization and Machine Learning | Gradient descent is the basic optimization procedure behind much of modern machine learning. It is simple enough to state in one line, but rich enough to expose many of the… |
| 14 | Chapter 14. Scientific Computing Applications | Differential equations are one of the main reasons automatic differentiation matters in scientific computing. Many scientific models are not written as closed-form functions…. |
| 15 | Chapter 15. Differentiable Systems Architecture | An end-to-end differentiable pipeline is a system whose final objective can send derivative information backward through every trainable or tunable stage of computation…. |
| 16 | Chapter 16. Sparse and Structured Differentiation | Sparse and structured differentiation studies how to compute derivatives without materializing dense derivative objects. Many real systems have enormous Jacobians and… |
| 17 | Chapter 17. Numerical and Systems Concerns | Automatic differentiation computes derivatives by executing arithmetic. On a real machine, arithmetic uses finite precision. This means AD gives the derivative of the… |
| 18 | Chapter 18. Advanced Topics | Many programs do not compute their output by applying a fixed sequence of explicit operations. Instead, they define the output as the solution of another problem. |
| 19 | Chapter 19. Theory and Foundations | Automatic differentiation is often described by a simple rule: |
| 20 | Chapter 20. Building an AD Engine | A minimal forward mode automatic differentiation engine has one job: evaluate a program while carrying both a value and its derivative. The engine does not build a graph. It… |
| 21 | Chapter 21. Major AD Systems | ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical… |
| 22 | Chapter 22. Open Problems | Automatic differentiation works naturally on pure mathematical functions: |
| Appendix | Appendix | ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical… |
Chapter 1. IntroductionA derivative measures how an output changes when an input changes. That sentence is simple, but it is one of the main ideas behind numerical computing, optimization, machine...
Chapter 2. Mathematical FoundationsAutomatic differentiation begins with a simple object: a function.
Chapter 3. Programs as Mathematical ObjectsA straight-line program is the simplest model of computation used in automatic differentiation. It is a program with a fixed sequence of assignments, no branches, no loops,...
Chapter 4. Core Theory of Automatic DifferentiationAutomatic differentiation is built on a simple observation: a complicated derivative can be computed by composing many small local derivatives. Instead of manipulating a full...
Chapter 5. Forward Mode Automatic DifferentiationForward mode automatic differentiation computes derivatives by carrying two values through a program at the same time: the ordinary value and its tangent. The ordinary value...
Chapter 6. Reverse Mode Automatic DifferentiationReverse mode automatic differentiation computes derivatives by propagating sensitivities backward through a computation. In forward mode, each intermediate value carries a...
Chapter 7. Dual Numbers and Algebraic StructuresDual numbers give the cleanest algebraic model of forward mode automatic differentiation. They extend ordinary real numbers with a formal infinitesimal part. Instead of...
Chapter 8. Higher-Order DifferentiationFirst derivatives describe local rate of change. Second derivatives describe how that rate of change itself changes. In optimization, this is curvature. In dynamics, it is...
Chapter 9. Differentiation of Control FlowA conditional is a program construct that chooses one computation among several possible computations. In ordinary code, this is written as if, else, switch, case, pattern...
Chapter 10. Matrix and Tensor DifferentiationMatrix calculus is the notation and rule system used to differentiate functions whose inputs, outputs, or intermediate values are vectors, matrices, or tensors. Automatic...
Chapter 11. Compiler and Runtime DesignSource transformation is an implementation strategy for automatic differentiation in which a program that computes a function is rewritten into another program that computes...
Chapter 12. AD in Modern Programming LanguagesLisp is one of the natural homes of automatic differentiation. It treats programs as data, has a simple expression syntax, and supports macro systems that can transform code...
Chapter 13. Optimization and Machine LearningGradient descent is the basic optimization procedure behind much of modern machine learning. It is simple enough to state in one line, but rich enough to expose many of the...
Chapter 14. Scientific Computing ApplicationsDifferential equations are one of the main reasons automatic differentiation matters in scientific computing. Many scientific models are not written as closed-form functions....
Chapter 15. Differentiable Systems ArchitectureAn end-to-end differentiable pipeline is a system whose final objective can send derivative information backward through every trainable or tunable stage of computation....
Chapter 16. Sparse and Structured DifferentiationSparse and structured differentiation studies how to compute derivatives without materializing dense derivative objects. Many real systems have enormous Jacobians and...
Chapter 17. Numerical and Systems ConcernsAutomatic differentiation computes derivatives by executing arithmetic. On a real machine, arithmetic uses finite precision. This means AD gives the derivative of the...
Chapter 18. Advanced TopicsMany programs do not compute their output by applying a fixed sequence of explicit operations. Instead, they define the output as the solution of another problem.
Chapter 19. Theory and FoundationsAutomatic differentiation is often described by a simple rule:
Chapter 20. Building an AD EngineA minimal forward mode automatic differentiation engine has one job: evaluate a program while carrying both a value and its derivative. The engine does not build a graph. It...
Chapter 21. Major AD SystemsADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical...
Chapter 22. Open ProblemsAutomatic differentiation works naturally on pure mathematical functions:
AppendixADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical...