Chapter 15. Differentiable Systems Architecture

End-to-End Differentiable Pipelines

An end-to-end differentiable pipeline is a system whose final objective can send derivative information backward through every trainable or tunable stage of computation. Instead of treating a model as the only differentiable component, the pipeline treats preprocessing, representation, simulation, retrieval, rendering, scoring, and decision rules as parts of one computational object.

The basic form is:

x \mapsto z_1 \mapsto z_2 \mapsto \cdots \mapsto z_k \mapsto y \mapsto L(y, t)

where $x$ is input data, $t$ is the target or constraint, $L$ is a scalar loss, and each stage $z_i$ is either differentiable or replaced by a differentiable approximation. Automatic differentiation computes how changes in parameters inside any stage affect the final loss.

Pipeline as Composition

A pipeline is a composition of functions:

F = f_k \circ f_{k-1} \circ \cdots \circ f_1

Each function may have its own parameters:

z_i = f_i(z_{i-1}; \theta_i)

The full parameter set is:

\theta = (\theta_1, \theta_2, \ldots, \theta_k)

The training objective is:

\min_\theta L(F(x; \theta), t)

Automatic differentiation applies the chain rule across the whole composition. For a parameter block $\theta_i$ , the derivative is:

\frac{\partial L}{\partial \theta_i} = \frac{\partial L}{\partial z_k} \frac{\partial z_k}{\partial z_{k-1}} \cdots \frac{\partial z_{i+1}}{\partial z_i} \frac{\partial z_i}{\partial \theta_i}

This expression is the mathematical reason end-to-end learning works. A late-stage loss can modify early-stage representations because the derivative path connects them.

Why End-to-End Differentiation Matters

Traditional systems often split work into hand-designed stages. For example, a vision system may contain image normalization, feature extraction, object detection, tracking, and final decision logic. Each stage is tuned separately.

A differentiable pipeline makes the final objective responsible for shaping the intermediate stages. This can remove mismatches between local objectives and system-level behavior.

For example, a feature extractor trained for classification may discard information needed for localization. If the whole detection pipeline is differentiable, the feature extractor receives gradients from the localization loss, not only from a classification loss.

The important shift is from local correctness to global usefulness.

Example: Differentiable Perception Pipeline

Consider a perception pipeline:

\text{image} \rightarrow \text{encoder} \rightarrow \text{features} \rightarrow \text{detector} \rightarrow \text{objects} \rightarrow \text{planner} \rightarrow \text{action} \rightarrow \text{loss}

If every stage is differentiable, the planning loss can affect the visual encoder. The encoder is no longer trained only to recognize objects. It is trained to produce information useful for action.

This matters when the final task depends on small details. A navigation policy may care less about object category and more about geometry, distance, and collision risk. End-to-end training allows those requirements to influence the earlier computation.

Interface Between Stages

A differentiable pipeline needs stable interfaces between stages. Each stage must expose values that support gradient propagation.

A minimal stage interface has the form:

forward(input, params) -> output
backward(output_grad, saved_state) -> input_grad, params_grad

In an AD system, the backward function may be generated automatically. The stage still has to obey certain constraints:

Requirement	Meaning
Differentiable computation	The forward pass must have usable derivatives almost everywhere
Saved intermediates	Reverse mode needs enough state to compute adjoints
Shape discipline	Tensor ranks and dimensions must remain consistent
Numerical stability	Gradients should avoid avoidable overflow, underflow, and cancellation
Clear parameter ownership	Each parameter must belong to a known stage
Controlled side effects	Mutation, randomness, and I/O must be handled explicitly

The interface does not need to expose symbolic formulas. It needs to expose enough structure for AD to replay or transform the computation.

Differentiable and Non-Differentiable Boundaries

Real pipelines contain non-differentiable operations. Examples include:

Operation	Issue
Sorting	Output changes discontinuously when order changes
Argmax	Selects one index and discards local sensitivity
Sampling	Random discrete choices break ordinary derivatives
Parsing	Often contains hard symbolic decisions
Database lookup	Returns records by exact key or predicate
Compression	Quantization loses continuous information
Rendering visibility	Occlusion changes discontinuously at boundaries

End-to-end differentiability does not require every operation to be smoothly differentiable in the classical sense. It requires a useful gradient signal.

Common strategies include smooth relaxation, straight-through estimators, surrogate losses, implicit differentiation, policy-gradient estimators, and custom adjoints.

Smooth Relaxation

A hard operation can often be replaced by a smooth approximation during training.

For example, argmax can be replaced by softmax:

\operatorname{softmax}(x_i) = \frac{\exp(x_i / \tau)} {\sum_j \exp(x_j / \tau)}

The temperature $\tau$ controls sharpness. Large $\tau$ gives a smooth distribution. Small $\tau$ approaches a hard choice.

This creates a differentiable path through what was previously a discrete selection. The trained system may still use the hard operation at inference time.

Custom Gradients

Some operations have forward behavior that is difficult to differentiate directly but have useful manually specified backward behavior.

A custom gradient defines:

y = op(x)

during backward:
    dx = custom_rule(x, y, dy)

This is common for numerical solvers, clipping operations, quantization, rendering, and specialized kernels.

Custom gradients are powerful but dangerous. They define the optimization behavior of the system. If the backward rule does not correspond to the forward computation, training may optimize a surrogate problem rather than the actual one.

State and Side Effects

End-to-end differentiable systems often interact with mutable state:

state_t -> system -> state_{t+1}

For a sequence of steps:

s_{t+1} = f(s_t, a_t; \theta)

the final loss may depend on the whole trajectory:

L = \sum_{t=0}^{T} \ell(s_t, a_t)

Reverse mode differentiates through time by propagating adjoints backward across the state transitions. This is the same structural idea as backpropagation through time.

The practical difficulty is memory. Reverse mode may need many intermediate states. Long pipelines require checkpointing, recomputation, truncation, or implicit gradient methods.

Pipeline Training Objective

A differentiable pipeline may contain multiple losses:

L = \lambda_1 L_1 + \lambda_2 L_2 + \cdots + \lambda_m L_m

Each loss may attach to a different stage.

For example:

encoder -> representation -> predictor -> decision
   |              |             |
contrastive    auxiliary     task loss
loss           loss

Auxiliary losses help early stages learn useful structure before the final task loss becomes informative. However, they also bias the pipeline. Poorly chosen auxiliary losses can conflict with the final objective.

The system designer must decide whether intermediate stages should be supervised directly or shaped only by the final loss.

End-to-End Does Not Mean One Monolith

A common mistake is to interpret end-to-end differentiability as a demand for one large unstructured model. That is unnecessary.

A good differentiable pipeline can be modular:

input
  -> differentiable parser
  -> typed representation
  -> neural or symbolic module
  -> differentiable solver
  -> calibrated output
  -> loss

The modules may have distinct types, invariants, parameter sets, and numerical methods. What matters is that the derivative path is defined across their boundaries.

Modularity remains important for testing, debugging, replacement, and interpretation.

Failure Modes

End-to-end differentiable pipelines fail in recognizable ways.

Failure mode	Cause
Gradient starvation	Early stages receive weak or noisy gradients
Shortcut learning	The system exploits accidental correlations
Unstable training	Long derivative chains amplify or suppress gradients
Memory blowup	Reverse mode stores too many intermediates
Invalid relaxation	Smooth surrogate differs too much from hard inference behavior
Poor credit assignment	Many stages influence the same loss ambiguously
Hidden nondeterminism	Randomness or concurrency makes gradients irreproducible
Boundary mismatch	Training uses differentiable approximations, inference uses hard operations

A pipeline may be differentiable but still hard to optimize. Differentiability gives a gradient. It does not guarantee that the gradient is useful.

Design Principles

A practical end-to-end differentiable pipeline should follow a few rules.

First, keep stage contracts explicit. Each module should define its input type, output type, parameter set, and differentiation behavior.

Second, separate training-time relaxations from inference-time behavior. If the system trains with a smooth proxy and runs with a hard operation, the mismatch must be measured.

Third, make gradient paths observable. Track gradient norms per stage. A silent zero gradient is a systems bug, not only a modeling issue.

Fourth, control memory deliberately. Long pipelines should be designed with checkpointing and recomputation in mind from the beginning.

Fifth, test local derivatives. Finite-difference checks, although slow and approximate, remain useful for validating custom adjoints and solver gradients.

Minimal Pseudocode

A simple end-to-end pipeline can be written as:

def pipeline(x, params):
    z1 = preprocess(x, params.preprocess)
    z2 = encode(z1, params.encoder)
    z3 = solve(z2, params.solver)
    y = decode(z3, params.decoder)
    return y

def objective(x, target, params):
    y = pipeline(x, params)
    return loss(y, target)

grads = autodiff.grad(objective)(x, target, params)

The important point is that preprocess, encode, solve, and decode are all inside the differentiated computation. If solve escapes into opaque code with no gradient rule, the derivative path stops there.

Systems View

End-to-end differentiable pipelines are not only mathematical objects. They are runtime systems.

They need:

Component	Purpose
Graph or trace representation	Records the differentiable computation
Runtime scheduler	Executes forward and backward passes
Memory planner	Stores or recomputes intermediate values
Kernel library	Provides efficient primitive operations
Custom gradient registry	Defines adjoints for special operations
Debugging tools	Inspect values, gradients, and graph structure
Serialization format	Saves model, parameters, and pipeline configuration

At small scale, an AD library can hide these concerns. At large scale, they become architectural constraints.

Core Idea

An end-to-end differentiable pipeline treats the whole system as a composed function with a trainable structure. Automatic differentiation supplies the derivative of the final objective with respect to parameters distributed throughout the pipeline.

The design problem is therefore twofold: define a computation that expresses the real task, and define derivative paths that provide useful credit assignment across all important stages.