Hyper-Dual Numbers

Dual numbers compute first derivatives exactly. Truncated polynomial algebras extend this to higher-order derivatives, but practical higher-order differentiation introduces an important problem: extracting second derivatives accurately without symbolic expansion or numerical cancellation.

Hyper-dual numbers solve this problem by introducing multiple nilpotent infinitesimal directions whose mixed products survive.

They provide an exact algebraic mechanism for computing:

• second derivatives
• mixed partial derivatives
• Hessians

without finite differences and without truncation error.

Motivation

Ordinary dual numbers satisfy:

Evaluating

produces:

Only first-order information survives.

To recover second derivatives, one possibility is nested dual numbers or truncated polynomial algebras. However, those approaches may:

• increase implementation complexity
• require managing higher polynomial coefficients
• introduce perturbation confusion in nested systems

Hyper-dual numbers provide a cleaner construction for exact second-order differentiation.

The Hyper-Dual Algebra

Introduce two independent infinitesimal generators:

Require:

But preserve the mixed product:

Also:

A hyper-dual number has the form:

This algebra stores:

Why Mixed Products Matter

The key idea is that:

The square does not vanish completely because cross terms survive.

This allows second-order information to appear algebraically.

Taylor Expansion

For a smooth scalar function:

the second-order Taylor expansion is:

Now substitute:

Since:

the square becomes:

Thus:

The coefficient of:

is exactly the second derivative.

Example

Let:

Use the hyper-dual input:

Expand:

First compute:

Since:

and:

while:

we obtain:

Thus:

Since:

the mixed coefficient gives the exact second derivative.

Multivariable Functions

Hyper-dual numbers naturally extend to multivariate functions.

Suppose:

Choose two perturbation directions:

Evaluate:

Then:

expands to:

The mixed coefficient gives the Hessian bilinear form:

This computes exact second-order directional derivatives.

Hessian Extraction

To compute a Hessian entry:

seed:

Then the coefficient of:

is exactly:

Repeated evaluation recovers the full Hessian matrix.

Example: Two Variables

Let:

Choose perturbations:

Then:

Mixed terms appear automatically.

Expanding the entire function produces coefficients involving:

which equal:

No symbolic differentiation is needed.

Exactness

Hyper-dual differentiation is exact up to floating-point arithmetic.

Unlike finite differences:

No step size is required.

No subtraction cancellation occurs.

The derivative structure emerges algebraically.

Algebraic Structure

The hyper-dual algebra can be written:

Basis elements are:

Dimension is four.

Multiplication rules:

This carefully chosen nilpotent structure isolates second-order interactions.

Computational Interpretation

A hyper-dual number may be represented as:

type HyperDual struct {
    Val  float64
    D1   float64
    D2   float64
    D12  float64
}

Components represent:

Multiplication Rule

Suppose:

and

Then multiplication becomes:

The mixed term obeys the second-order product rule automatically.

Example Implementation

func Mul(x, y HyperDual) HyperDual {
    return HyperDual{
        Val: x.Val * y.Val,

        D1:
            x.D1*y.Val +
            x.Val*y.D1,

        D2:
            x.D2*y.Val +
            x.Val*y.D2,

        D12:
            x.D12*y.Val +
            x.D1*y.D2 +
            x.D2*y.D1 +
            x.Val*y.D12,
    }
}

The D12 component contains all mixed second-order interactions.

Relation to Hessian-Vector Products

Hyper-dual numbers compute second-order directional derivatives naturally.

Given:

evaluate:

The coefficient of:

is the result.

This avoids explicit Hessian construction.

For large systems, Hessian-vector products are often preferable to dense Hessians.

Perturbation Confusion

Nested dual-number systems may accidentally mix perturbation symbols.

Hyper-dual numbers avoid this by explicitly separating infinitesimal generators:

Each perturbation direction remains algebraically distinct.

This improves correctness in higher-order implementations.

Relation to Truncated Polynomial Algebras

Hyper-dual numbers differ from ordinary truncated polynomial algebras.

Truncated polynomial algebra:

keeps powers:

Hyper-dual algebra instead keeps:

This distinction matters:

Hyper-dual systems are particularly effective for Hessian computation.

Complexity

For $n$ variables:

• one forward dual pass computes one directional derivative
• one hyper-dual pass computes one second-order directional interaction

Dense Hessian construction still requires multiple evaluations.

However, the method remains exact and compositional.

Geometric Interpretation

Dual numbers represent tangent vectors.

Hyper-dual numbers represent interacting tangent directions.

The mixed product:

captures curvature.

First-order infinitesimals describe local linear geometry.

Second-order mixed infinitesimals describe local quadratic geometry.

Hyper-dual numbers therefore encode second-order local structure.

Summary

Hyper-dual numbers extend dual numbers by introducing multiple independent nilpotent directions whose mixed products survive.

The algebra:

produces exact second derivatives through ordinary program evaluation.

Key properties:

Hyper-dual numbers provide one of the cleanest exact formulations of second-order automatic differentiation.