ADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical...
| Section | Title |
|---|---|
| 1 | Appendix |
| 2 | Tapenade |
| 3 | TensorFlow Autograd |
| 4 | PyTorch Autograd |
| 5 | JAX |
| 6 | Zygote |
| 7 | Enzyme |
| 8 | Tinygrad |
| 9 | Comparative Architecture Analysis |
| 10 | Differentiation of Large Stateful Systems |
AppendixADIFOR, short for Automatic Differentiation of Fortran, is one of the classical source-transformation systems for automatic differentiation. It was designed for numerical...
TapenadeTapenade is a source-transformation automatic differentiation system developed at INRIA. Like ADIFOR, it takes an existing program and produces a new differentiated program....
TensorFlow AutogradTensorFlow Autograd refers to TensorFlow’s automatic differentiation system, mainly exposed through tf.GradientTape. It is a reverse-mode AD system designed for tensor...
PyTorch AutogradPyTorch Autograd is a dynamic reverse-mode automatic differentiation system. It records tensor operations as they execute, builds a computation graph at runtime, and then...
JAXJAX is an automatic differentiation and array programming system for Python. It combines NumPy-like syntax with composable program transformations. Its core transformations...
ZygoteZygote is a source-to-source reverse-mode automatic differentiation system for the Julia programming language. It was designed to differentiate high-level Julia code directly,...
EnzymeEnzyme is a compiler-based automatic differentiation system for LLVM and MLIR. Instead of differentiating source code directly, or recording tensor operations at runtime,...
TinygradTinygrad is a small deep learning framework centered around a minimal reverse-mode automatic differentiation engine. It was created by entity"people","George...
Comparative Architecture AnalysisThe systems in this chapter show that automatic differentiation is not one implementation technique. It is a family of program transformations. Each system chooses a different...
Differentiation of Large Stateful SystemsAutomatic differentiation works naturally on pure mathematical functions: