# Differentiable Rendering

## Differentiable Rendering

Differentiable rendering is the process of computing derivatives of rendered images with respect to scene parameters. A renderer becomes part of the computational graph rather than a terminal visualization step.

The central mapping is:

$$
\theta \mapsto I
$$

where:

| Symbol | Meaning |
|---|---|
| $\theta$ | Scene parameters |
| $I$ | Rendered image |

The parameters may include:

- geometry
- camera pose
- material properties
- lighting
- texture maps
- volumetric density
- skeletal pose
- simulation state

A loss compares the rendered image against observations:

$$
L(I, I^\star)
$$

Automatic differentiation computes:

$$
\frac{\partial L}{\partial \theta}
$$

allowing optimization of scene parameters directly from pixels.

### Rendering as a Function

A renderer can be modeled as:

$$
I = R(S, C, M, L)
$$

where:

| Variable | Meaning |
|---|---|
| $S$ | Scene geometry |
| $C$ | Camera parameters |
| $M$ | Material model |
| $L$ | Lighting configuration |

Differentiable rendering computes gradients through this mapping.

Traditional rendering solves:

```text
scene -> image
```

Differentiable rendering solves the inverse problem:

```text
image -> scene parameters
```

through optimization.

### Why Differentiable Rendering Matters

Classical graphics pipelines are forward simulators. They generate images from known scenes.

Many real problems require recovering the scene itself:

| Problem | Unknown Quantity |
|---|---|
| 3D reconstruction | Geometry |
| Camera calibration | Pose and intrinsics |
| Inverse rendering | Materials and lights |
| Motion capture | Skeleton parameters |
| Robotics perception | Object shape and pose |
| Medical imaging | Internal structure |
| Neural avatars | Dynamic appearance |
| Scientific simulation | Physical parameters |

Differentiable rendering allows these problems to be optimized using gradient descent.

### Image Formation

A renderer approximates light transport.

The rendering equation is:

$$
L_o(x,\omega_o)=L_e(x,\omega_o)+\int_{\Omega} f_r(x,\omega_i,\omega_o)L_i(x,\omega_i)(n\cdot\omega_i)d\omega_i
$$

where:

| Symbol | Meaning |
|---|---|
| $L_o$ | Outgoing radiance |
| $L_e$ | Emitted radiance |
| $f_r$ | Bidirectional reflectance distribution |
| $L_i$ | Incoming light |
| $n$ | Surface normal |
| $\omega_i$ | Incoming direction |
| $\omega_o$ | Outgoing direction |

Differentiable rendering computes derivatives of image intensity with respect to scene variables embedded inside this equation.

### Computational Graph of Rendering

A rasterization pipeline may look like:

```text
vertices
  -> transforms
  -> projection
  -> rasterization
  -> shading
  -> compositing
  -> image
```

The backward pass propagates gradients through every stage.

For example:

$$
\frac{\partial L}{\partial v_i}
$$

tells how moving vertex $v_i$ changes the image loss.

This enables direct optimization of geometry from visual supervision.

### Rasterization and Discontinuity

Classical rasterization is discontinuous.

A tiny vertex movement may suddenly:

- expose a triangle
- hide a triangle
- change visibility
- switch pixel ownership

This creates undefined or unstable derivatives.

The main challenge of differentiable rendering is therefore visibility discontinuity.

### Soft Rasterization

Soft rasterization replaces hard visibility decisions with smooth approximations.

Instead of:

```text
pixel belongs to triangle T
```

the renderer computes probabilities:

$$
p_{ij} = \operatorname{soft_visibility}(p_i, T_j)
$$

The pixel color becomes:

$$
C_i = \sum_j p_{ij} c_{ij}
$$

where:

| Variable | Meaning |
|---|---|
| $p_{ij}$ | Probability triangle $j$ contributes |
| $c_{ij}$ | Triangle color contribution |

This creates usable gradients near edges and occlusion boundaries.

### Differentiable Ray Tracing

Ray tracing simulates light transport through ray intersections.

A ray:

$$
r(t) = o + td
$$

intersects scene geometry.

Differentiating ray tracing is difficult because intersections change discontinuously when geometry moves.

Approaches include:

| Method | Strategy |
|---|---|
| Path-space differentiation | Differentiate full light paths |
| Edge sampling | Estimate visibility gradients |
| Soft visibility | Smooth occlusion |
| Monte Carlo estimators | Stochastic gradient estimation |
| Reparameterization | Stabilize sampling derivatives |

Differentiable Monte Carlo rendering is especially important in physically based inverse rendering.

### Geometry Optimization

Suppose a mesh has vertices:

$$
V = \{v_1, \ldots, v_n\}
$$

The renderer produces:

$$
I = R(V)
$$

Given a target image $I^\star$:

$$
L = \|I - I^\star\|^2
$$

The gradient:

$$
\frac{\partial L}{\partial v_i}
$$

moves vertices toward shapes that better explain the observation.

This enables:

- mesh fitting
- shape reconstruction
- pose estimation
- registration
- scene alignment

### Camera Optimization

Camera parameters are also differentiable.

Projection often has the form:

$$
p =
K [R|t] X
$$

where:

| Symbol | Meaning |
|---|---|
| $K$ | Intrinsic matrix |
| $R$ | Rotation |
| $t$ | Translation |
| $X$ | 3D point |

Differentiable rendering computes gradients with respect to:

- focal length
- distortion
- orientation
- translation
- projection parameters

This supports bundle adjustment and self-calibration.

### Material Optimization

Materials determine surface appearance.

A material model may contain:

- albedo
- roughness
- metallic properties
- subsurface scattering
- refractive index

The renderer computes:

$$
I = R(M)
$$

and gradients:

$$
\frac{\partial L}{\partial M}
$$

allow optimization of appearance parameters from images.

Inverse material estimation is fundamental in digital asset reconstruction.

### Volumetric Rendering

Many modern differentiable renderers are volumetric.

Instead of surfaces, scenes are represented as density fields:

$$
\sigma(x)
$$

and color fields:

$$
c(x, d)
$$

A ray accumulates contributions along its path:

$$
C(r) =
\int T(t)\sigma(r(t))c(r(t), d)dt
$$

where $T(t)$ is transmittance.

This formulation is naturally differentiable because accumulation is continuous.

Neural radiance fields use this structure extensively.

### Neural Rendering

Neural rendering replaces parts of the graphics pipeline with learned functions.

Example pipeline:

```text
scene parameters
  -> neural field
  -> differentiable renderer
  -> image
```

The neural field may represent:

- density
- color
- material response
- deformation
- lighting

Optimization occurs jointly over neural weights and scene parameters.

### Neural Radiance Fields

NeRF-like systems model a scene as:

$$
F_\theta(x, d)
=
(\sigma, c)
$$

where:

| Symbol | Meaning |
|---|---|
| $x$ | Spatial coordinate |
| $d$ | Viewing direction |
| $\sigma$ | Density |
| $c$ | Color |

Rendering integrates these values along rays.

The whole process is differentiable with respect to network parameters $\theta$.

This allows 3D scene reconstruction directly from posed images.

### Temporal Differentiable Rendering

Dynamic scenes introduce time:

$$
I_t = R(S_t)
$$

Gradients may propagate through:

- skeletal animation
- deformation fields
- physical simulation
- fluid motion
- camera trajectories

The system becomes a spatiotemporal differentiable simulator.

### Differentiable Physics and Rendering

Many pipelines combine simulation and rendering:

```text
physics simulation
  -> scene state
  -> renderer
  -> image loss
```

Gradients may flow from image observations back into physical parameters:

- mass
- friction
- elasticity
- force fields
- control signals

This connects computer graphics, robotics, and scientific inference.

### Loss Functions

Differentiable rendering rarely optimizes raw pixels alone.

Common losses include:

| Loss | Purpose |
|---|---|
| Pixel MSE | Direct reconstruction |
| Perceptual loss | Feature similarity |
| Silhouette loss | Shape alignment |
| Depth loss | Geometric consistency |
| Normal loss | Surface orientation |
| Adversarial loss | Realistic appearance |
| Multi-view consistency | Cross-camera agreement |

Loss design strongly affects reconstruction quality.

### Monte Carlo Gradient Noise

Physically based rendering often uses stochastic sampling:

$$
I \approx \frac{1}{N}\sum_i f(x_i)
$$

Differentiating stochastic estimates introduces variance.

Problems include:

| Issue | Effect |
|---|---|
| High variance gradients | Slow optimization |
| Visibility discontinuities | Unstable updates |
| Sparse lighting paths | Weak signals |
| Sampling bias | Incorrect gradients |

Variance reduction is therefore central in differentiable rendering research.

### Memory and Performance

Differentiable rendering is computationally expensive.

The backward pass may require:

- geometry buffers
- visibility information
- sampled paths
- intermediate shading state
- volumetric accumulations

Memory costs grow quickly with:

- image resolution
- ray count
- scene complexity
- temporal length

Large systems rely on:

| Technique | Purpose |
|---|---|
| Checkpointing | Reduce stored state |
| Recomputation | Trade compute for memory |
| Mixed precision | Reduce bandwidth |
| Sparse gradients | Limit updates |
| Hierarchical acceleration | Reduce ray cost |

### Hybrid Rendering Systems

Practical systems often combine symbolic and neural components.

Example:

```text
mesh geometry
  -> rasterization
  -> neural shading
  -> compositing
  -> image
```

or:

```text
physics engine
  -> differentiable renderer
  -> learned policy
```

Hybrid systems are usually more stable and interpretable than fully learned renderers.

### Failure Modes

Differentiable rendering systems fail in characteristic ways.

| Failure | Cause |
|---|---|
| Geometry collapse | Weak shape supervision |
| Texture baking | Appearance memorized into textures |
| Floaters | Density artifacts in volumetric fields |
| Gradient spikes | Visibility discontinuities |
| Over-smoothed geometry | Soft rasterization bias |
| Ambiguous reconstruction | Multiple scenes explain same image |
| Lighting-shape confusion | Ill-posed inverse problem |

Many inverse rendering problems are fundamentally underdetermined.

### Systems Architecture

A differentiable rendering engine typically contains:

| Component | Purpose |
|---|---|
| Scene graph | Represents geometry and materials |
| Acceleration structures | Fast visibility queries |
| Rasterizer or ray tracer | Image generation |
| Differentiable runtime | Gradient propagation |
| Tensor backend | GPU computation |
| Sampling engine | Monte Carlo integration |
| Optimization loop | Parameter updates |
| Caching system | Reuse expensive computations |

Modern systems increasingly integrate tightly with machine learning runtimes.

### Relation to Automatic Differentiation

Differentiable rendering extends automatic differentiation into graphics and physical image formation.

The renderer becomes another differentiable operator:

$$
R : \Theta \to \mathbb{R}^{H \times W \times C}
$$

Automatic differentiation propagates gradients through:

- geometry
- visibility approximations
- shading
- light transport
- volumetric integration
- neural scene representations

The main difficulty is not algebraic differentiation itself. The difficulty is handling discontinuous visibility, stochastic transport, and large-scale geometric computation while maintaining useful gradients.

### Core Idea

Differentiable rendering transforms image synthesis into an optimization-compatible process. Instead of using rendering only to generate pictures, the renderer becomes a bridge between visual observations and latent scene structure.

Automatic differentiation allows errors measured in image space to shape geometry, materials, motion, lighting, and physical parameters throughout the scene representation.