brain

A clear explanation of counting unique pairs whose absolute difference is k using frequency counting.

A clear graph traversal solution for finding all processes terminated when killing a target process.

A clear linear-time solution for finding the shortest subarray that must be sorted to make the whole array sorted.

A clear explanation of Next Greater Element III using the next permutation algorithm on the digits of an integer.

A clear explanation of counting black pixels that are alone in both their row and column.

A clear explanation of Split Concatenated Strings using string reversal choices and enumeration of every possible cut point.

A clear SQL guide for counting students in every department, including departments with zero students.

A clear explanation of finding the minimum difference between two BST node values using inorder traversal.

A clear SQL guide for computing each employee's 3-month cumulative salary while excluding their most recent month.

A clear explanation of Brick Wall using prefix sums and a hash map to find the best vertical cut position.

A clear explanation of updating a Minesweeper board using DFS flood fill and adjacent mine counting.

A clear explanation of Optimal Division using the structure of division expressions to build the maximum-value expression.

A clear SQL guide for finding the question with the highest answer rate from survey logs.

A clear explanation of weighted random sampling using prefix sums and binary search.

A clear explanation of Student Attendance Record II using dynamic programming over absence count and late streak.

A clear SQL guide for finding employees whose bonus is less than 1000 or missing.

A clear explanation of generating minimal unique word abbreviations using grouping and trie prefixes.

A clear dynamic programming solution for counting paths that move a ball out of a grid boundary.

A clear explanation of Student Attendance Record I using simple string checks and a one-pass counter solution.

A clear explanation of counting beautiful arrangements using backtracking and divisibility pruning.

Arithmetic geometry studies solutions of polynomial equations by combining algebra, geometry, and number theory. Its basic objects are spaces defined by polynomial equations....

A field is a number system in which addition, subtraction, multiplication, and division by nonzero elements are always possible. The rational numbers $\mathbb{Q}$, the real...

The harmonic series is the infinite series

A Diophantine equation is an equation whose solutions are required to be integers. The unknowns are not allowed to range over the real numbers or complex numbers unless...

Division of integers does not always produce an integer. For example,

The idea of number arose long before formal mathematics. Early civilizations used numbers for counting objects, measuring land, recording trade, and tracking time.

The integers extend infinitely in both directions:

Many mathematical objects are defined recursively. A recursive definition specifies:

Ordinary induction proves a statement $Pn$ by showing that truth passes from one case to the next:

Many statements in number theory concern all natural numbers. For example, one may wish to prove that

The order relation distinguishes positive and negative integers, but in many situations the sign of a number is less important than its magnitude. For example, the integers

The integers are not merely a collection of numbers equipped with arithmetic operations. They also possess an order structure. Given two integers $a$ and $b$, one can...

An arithmetic operation is a rule that combines numbers to produce another number. The most basic operations on integers are addition, subtraction, multiplication, and division.

The natural numbers are sufficient for counting and addition, but they are not sufficient for subtraction. For example,

The natural numbers arise from the basic act of counting. When we count objects in a collection, we assign successive numbers:

Automatic differentiation works naturally on pure mathematical functions:

Automatic differentiation works naturally on pure mathematical functions:

Automatic differentiation works naturally on pure mathematical functions:

A minimal automatic differentiation engine can compute correct gradients on small programs. A production system must survive long-running workloads, large tensors, distributed...

Automatic differentiation began as a numerical technique for computing gradients of scalar functions.

Sparse and structured differentiation studies how to compute derivatives without materializing dense derivative objects. Many real systems have enormous Jacobians and...

Physics-informed models combine data fitting with equations from physics or applied mathematics. The model is trained not only to match observed samples, but also to satisfy...

Automatic differentiation is easiest to define for pure functions. A pure function behaves like a mathematical mapping: it consumes inputs, produces outputs, and has no...

Automatic differentiation can be performed before a program runs, while it runs, or in a staged phase between the two.

This section studies reverse mode automatic differentiation through concrete examples. Each case has the same structure:

Forward mode automatic differentiation computes derivatives by propagating tangent values alongside ordinary values. The ordinary value is called the primal. The derivative...

Differentiation describes how a function changes locally. A Taylor expansion extends this idea by approximating a function with a polynomial around a point.

The systems in this chapter show that automatic differentiation is not one implementation technique. It is a family of program transformations. Each system chooses a different...

The systems in this chapter show that automatic differentiation is not one implementation technique. It is a family of program transformations. Each system chooses a different...

An automatic differentiation engine is only useful if its derivatives are correct. A small mistake in a backward rule can silently corrupt optimization, training, or...

The preceding sections described automatic differentiation through algebraic, categorical, logical, and denotational models. These viewpoints converge on one central idea:

Automatic differentiation systems are usually trusted because they implement mathematically established rules such as the chain rule, product rule, and linearization of...

Distributed gradient computation appears when a differentiable program no longer fits comfortably on one device or one machine. The reason may be model size, data volume,...

Differentiable systems architecture extends automatic differentiation beyond isolated functions and neural network layers. The central idea is to treat larger systems as...

Probabilistic programming represents uncertainty using executable probabilistic models. A probabilistic program defines a distribution rather than only a deterministic computation.

Reinforcement learning studies learning systems that act in an environment. Unlike supervised learning, the training signal is not a target label for each input. The model...

Automatic differentiation interacts deeply with type systems because differentiation changes the structure of computation. A derivative operator maps one function into another...

Kernel fusion combines several small operations into one larger executable unit.

Modern automatic differentiation systems are fundamentally tensor compiler systems. Their performance depends less on mathematical differentiation rules than on how...

Higher-order automatic differentiation faces a fundamental problem: derivative structure grows combinatorially with order.

Automatic differentiation is deeply connected to functional programming and lambda calculus. Programs can be viewed as mathematical functions, and differentiation can be...

Reverse mode automatic differentiation is the mathematical and systems basis of backpropagation. In deep learning, the objective is usually a scalar loss depending on many...

Forward mode automatic differentiation appears in many numerical systems where directional derivatives, local sensitivities, or small parameter sets are important. This...

Automatic differentiation is a method for computing derivatives by transforming programs into derivative-propagating computations. It does not approximate derivatives...

Tinygrad is a small deep learning framework centered around a minimal reverse-mode automatic differentiation engine. It was created by entity"people","George...

Tinygrad is a small deep learning framework centered around a minimal reverse-mode automatic differentiation engine. It was created by entity"people","George...

Performance benchmarking measures whether an automatic differentiation engine is fast, memory-efficient, and scalable under realistic workloads. It also protects the engine...

Operational semantics explains how automatic differentiation executes. Denotational semantics explains what differentiable programs mean.

Automatic differentiation began as a transformation applied to numerical programs. A differentiable programming language instead treats differentiation as a native semantic...

Modern automatic differentiation systems are built around accelerator hardware. GPUs and TPUs provide enormous throughput for tensor operations, making large-scale...

A hybrid symbolic-numeric system combines discrete symbolic reasoning with continuous numerical computation. In the context of automatic differentiation, it means a pipeline...

Robotics and control systems interact with the physical world through sensing, estimation, planning, and actuation. Automatic differentiation is important because modern...

Meta-learning studies systems that improve how they learn. Instead of only optimizing model parameters for one task, a meta-learning method optimizes some part of the learning...

Swift became an important experiment in language-integrated automatic differentiation because it attempted to make differentiation a core compiler feature rather than a...

Memory planning determines where values are stored, how long they remain alive, and when storage can be reused.

Most real computational problems are sparse. Large matrices and tensors often contain mostly zeros, structured blocks, or local interactions. Sparse representations reduce...

Programs do not only branch between valid computations. They also fail, stop early, raise exceptions, return sentinel values, or enter undefined numerical regions. These...

Perturbation confusion is a correctness bug that appears in nested automatic differentiation, especially nested forward mode. It happens when two derivative computations...

Automatic differentiation is deeply connected to functional programming and lambda calculus. Programs can be viewed as mathematical functions, and differentiation can be...

Checkpointing is a technique for reducing the memory cost of reverse mode automatic differentiation by selectively storing intermediate states and recomputing missing values...

Many real-world Jacobians are sparse. Most derivative entries are zero because outputs depend only on small subsets of inputs.

Automatic differentiation can be understood as a transformation from one program into another program.

A differentiable subprogram is a program fragment that can participate in derivative propagation as a coherent unit. Instead of differentiating an entire application...

Enzyme is a compiler-based automatic differentiation system for LLVM and MLIR. Instead of differentiating source code directly, or recording tensor operations at runtime,...

Enzyme is a compiler-based automatic differentiation system for LLVM and MLIR. Instead of differentiating source code directly, or recording tensor operations at runtime,...

A custom gradient gives the user direct control over the backward rule of an operation. The forward computation still produces an ordinary value, but the derivative no longer...

Automatic differentiation systems are trusted infrastructure. Scientific computing, machine learning, optimization, simulation, and control systems depend on gradients being...

Quantum computation introduces a computational model fundamentally different from classical programs.

Automatic differentiation is usually described as a transformation of programs or computational graphs. In real systems, it is also a parallel execution problem. Large...

A differentiable operating system is an execution environment whose resource-management decisions can be optimized using gradients or gradient-like feedback. Instead of...

Signal processing studies how information is represented, transformed, filtered, compressed, reconstructed, and estimated from signals. A signal may be a time series, an...

An implicit layer defines its output as the solution of an equation, not as a fixed sequence of explicit operations. Instead of computing

Julia was designed for high-performance technical computing. It combines interactive syntax with a compiler capable of specializing code aggressively based on types. This...

Staging is the separation of a program into phases.

The singular value decomposition SVD is one of the most important matrix factorizations in numerical linear algebra. It appears in dimensionality reduction, least squares,...

A stateful system is a program whose output depends not only on its explicit inputs, but also on stored state. The state may live in variables, objects, arrays, files, random...

Higher-order derivatives contain rich geometric information, but naïve computation quickly becomes impractical.

Automatic differentiation can be described operationally through dual numbers and computational graphs. It can also be described abstractly using category theory.

Reverse mode automatic differentiation is computationally efficient for scalar-output functions, but it has a major systems cost: it needs information from the forward pass...

Forward mode automatic differentiation computes Jacobian-vector products: