Chapter 98. Tensor Products

Tensor products provide a systematic way to combine vector spaces into a larger space that encodes bilinear structure. They appear throughout modern mathematics, physics, numerical analysis, differential geometry, representation theory, and machine learning.

The tensor product transforms multilinear problems into linear problems. This reduction is the central reason for its importance.

If ordinary vectors describe quantities with one index, tensors describe quantities with multiple indices. Tensor products give the algebraic framework underlying this idea.

98.1 Motivation

Suppose $V$ and $W$ are vector spaces over a field $F$ .

We often encounter maps

B : V \times W \to U

that are linear in each variable separately.

Such maps are called bilinear.

For example, the dot product

\mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}

is bilinear because

(u + v) \cdot w = u \cdot w + v \cdot w

and

(cu) \cdot w = c(u \cdot w).

The same properties hold in the second argument.

The tensor product constructs a vector space

V \otimes W

such that every bilinear map from $V \times W$ factors uniquely through a linear map from $V \otimes W$ .

This converts bilinear algebra into ordinary linear algebra.

98.2 Bilinear Maps

A map

B : V \times W \to U

is bilinear if

B(v_1 + v_2, w) = B(v_1,w) + B(v_2,w)

and

B(cv,w) = cB(v,w)

for all $v,v_1,v_2 \in V$ , $w \in W$ , and scalars $c$ .

Similarly,

B(v,w_1+w_2) = B(v,w_1)+B(v,w_2)

and

B(v,cw) = cB(v,w).

Thus bilinearity means linearity in each argument independently.

Examples include:

Bilinear map	Domain	Codomain
Dot product	$\mathbb{R}^n \times \mathbb{R}^n$	$\mathbb{R}$
Matrix multiplication	$M_{m,n} \times M_{n,p}$	$M_{m,p}$
Polynomial multiplication	$P_m \times P_n$	$P_{m+n}$
Inner product	$V \times V$	$F$

Bilinear maps arise naturally whenever two independent linear inputs interact.

98.3 Definition of the Tensor Product

The tensor product of $V$ and $W$ is a vector space $V \otimes W$ together with a bilinear map

\tau : V \times W \to V \otimes W

satisfying the following universal property:

For every vector space $U$ and every bilinear map

B : V \times W \to U,

there exists a unique linear map

\widetilde{B} : V \otimes W \to U

such that

B(v,w) = \widetilde{B}(\tau(v,w)).

This definition may be represented by the commutative diagram

V \times W \overset{\tau}{\longrightarrow} V \otimes W \overset{\widetilde{B}}{\longrightarrow} U.

The composition equals the original bilinear map.

The tensor product therefore represents all bilinear maps simultaneously.

98.4 Pure Tensors

The image of a pair $(v,w)$ under $\tau$ is written

v \otimes w.

Such elements are called pure tensors or simple tensors.

The tensor product space is generated by pure tensors.

The bilinearity conditions imply:

(v_1+v_2)\otimes w = v_1\otimes w + v_2\otimes w,

v\otimes(w_1+w_2) = v\otimes w_1 + v\otimes w_2,

and

(cv)\otimes w = v\otimes(cw) = c(v\otimes w).

These identities define the algebraic structure of tensor products.

A general tensor is usually a sum of pure tensors:

x = \sum_{i=1}^k v_i \otimes w_i.

Not every tensor is itself pure.

98.5 Construction of the Tensor Product

The tensor product can be constructed explicitly.

Start with the free vector space generated by all pairs $(v,w)$ . Then impose the bilinearity relations:

(v_1+v_2,w) \sim (v_1,w)+(v_2,w),

(v,w_1+w_2) \sim (v,w_1)+(v,w_2),

(cv,w) \sim (v,cw).

The quotient space obtained after imposing these relations is $V \otimes W$ .

This construction ensures that bilinearity is built directly into the space itself.

98.6 Basis of a Tensor Product

Suppose

\{e_1,\ldots,e_m\}

is a basis for $V$ , and

\{f_1,\ldots,f_n\}

is a basis for $W$ .

Then

\{ e_i \otimes f_j : 1 \le i \le m, 1 \le j \le n \}

is a basis for $V \otimes W$ .

Therefore,

\dim(V \otimes W) = \dim(V)\dim(W).

If $\dim(V)=m$ and $\dim(W)=n$ , then

\dim(V\otimes W)=mn.

This resembles the size of Cartesian products, but tensor products encode linear structure rather than ordered pairs.

98.7 Coordinates in Tensor Products

Suppose

v = \sum_i a_i e_i, \qquad w = \sum_j b_j f_j.

Then

v\otimes w = \sum_{i,j} a_i b_j (e_i\otimes f_j).

The coefficients multiply because of bilinearity.

This behavior explains why tensors naturally represent multidimensional arrays.

For example, if

v= \begin{bmatrix} a_1 \\ a_2 \end{bmatrix}, \qquad w= \begin{bmatrix} b_1 \\ b_2 \end{bmatrix},

then

v\otimes w = \begin{bmatrix} a_1b_1 \\ a_1b_2 \\ a_2b_1 \\ a_2b_2 \end{bmatrix}.

The tensor product records all pairwise products of coordinates.

98.8 Tensor Product of Linear Maps

Suppose

T : V \to V'

and

S : W \to W'

are linear maps.

The tensor product map

T \otimes S : V\otimes W \to V'\otimes W'

is defined by

(T\otimes S)(v\otimes w) = T(v)\otimes S(w).

This definition extends linearly to all tensors.

Tensor products therefore preserve linear structure at the level of transformations as well as spaces.

If matrices represent $T$ and $S$ , then the matrix representing $T\otimes S$ is the Kronecker product of the matrices.

98.9 Kronecker Products

A= \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}

and

B= \begin{bmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{bmatrix},

then the Kronecker product is

A\otimes B = \begin{bmatrix} a_{11}B & a_{12}B \\ a_{21}B & a_{22}B \end{bmatrix}.

Explicitly,

A\otimes B = \begin{bmatrix} a_{11}b_{11} & a_{11}b_{12} & a_{12}b_{11} & a_{12}b_{12} \\ a_{11}b_{21} & a_{11}b_{22} & a_{12}b_{21} & a_{12}b_{22} \\ a_{21}b_{11} & a_{21}b_{12} & a_{22}b_{11} & a_{22}b_{12} \\ a_{21}b_{21} & a_{21}b_{22} & a_{22}b_{21} & a_{22}b_{22} \end{bmatrix}.

Kronecker products appear in signal processing, quantum mechanics, numerical PDEs, and tensor computation.

98.10 Tensor Rank

A tensor has rank $1$ if it is pure:

x=v\otimes w.

The rank of a tensor is the smallest number of pure tensors needed to express it:

x = \sum_{i=1}^r v_i\otimes w_i.

Tensor rank generalizes matrix rank.

Unlike matrix rank, tensor rank is often difficult to compute. Many problems involving tensor rank are computationally hard.

Low-rank tensor approximation is important in machine learning and scientific computing because it compresses large multidimensional datasets.

98.11 Dual Spaces and Tensor Products

Let $V^*$ denote the dual space of $V$ .

The tensor product

V^* \otimes W

can be identified with the space of linear maps

V \to W.

f\in V^*, \qquad w\in W,

then the tensor

f\otimes w

acts on $v\in V$ by

(f\otimes w)(v) = f(v)w.

This is a rank-one linear operator.

More generally,

V^*\otimes W \cong \operatorname{Hom}(V,W).

This identification is fundamental throughout multilinear algebra.

98.12 Symmetric and Alternating Tensors

Tensor products contain important subspaces.

A tensor is symmetric if swapping indices does not change it.

For example,

v\otimes w + w\otimes v

is symmetric.

A tensor is alternating if swapping indices changes the sign.

For example,

v\otimes w - w\otimes v

is alternating.

These ideas lead to:

Structure	Associated algebra
Symmetric tensors	Symmetric algebra
Alternating tensors	Exterior algebra
Mixed tensors	General tensor algebra

Alternating tensors are central in differential geometry and topology.

98.13 Tensor Algebra

The tensor algebra of $V$ is

T(V) = \bigoplus_{k=0}^{\infty} V^{\otimes k}.

Here

V^{\otimes k} = \underbrace{ V\otimes \cdots \otimes V }_{k\text{ times}}.

The tensor algebra contains tensors of every order.

Elements of:

Space	Interpretation
$V$	Vectors
$V\otimes V$	Second-order tensors
$V^{\otimes 3}$	Third-order tensors
$V^{\otimes k}$	$k$ -tensors

Tensor algebras provide the foundation for exterior algebras, Clifford algebras, and representation theory.

98.14 Tensors in Physics

Tensors describe physical quantities that remain meaningful under coordinate transformations.

Examples include:

Tensor	Physical meaning
Stress tensor	Internal forces in materials
Metric tensor	Geometry of spacetime
Electromagnetic tensor	Electric and magnetic fields
Inertia tensor	Rotational dynamics
Curvature tensor	Spacetime curvature

In relativity, tensors encode laws of physics independently of coordinates.

This coordinate-independent formulation is one of the major achievements of tensor calculus.

98.15 Tensors in Machine Learning

Modern machine learning frequently represents data as tensors.

Examples include:

Data type	Tensor order
Vector	1
Matrix	2
RGB image	3
Video	4
Transformer attention weights	Higher-order

Tensor decompositions generalize matrix factorizations such as the singular value decomposition.

Popular decompositions include:

Decomposition	Purpose
CP decomposition	Rank decomposition
Tucker decomposition	Multilinear compression
Tensor train	Efficient high-dimensional representation

Large-scale tensor computation is fundamental in deep learning systems.

98.16 Universal Properties

The universal property defines tensor products abstractly and uniquely.

This viewpoint is important because it avoids dependence on coordinates or explicit constructions.

Many advanced mathematical structures are defined by universal properties.

The tensor product is one of the simplest and most important examples.

The key principle is:

Bilinear maps from $V\times W$ correspond exactly to linear maps from $V\otimes W$ .

This correspondence is natural and canonical.

98.17 Example

Let

V=W=\mathbb{R}^2

with standard basis

e_1= \begin{bmatrix} 1 \\ 0 \end{bmatrix}, \qquad e_2= \begin{bmatrix} 0 \\ 1 \end{bmatrix}.

Then

V\otimes W

has basis

\{ e_1\otimes e_1, e_1\otimes e_2, e_2\otimes e_1, e_2\otimes e_2 \}.

Hence

\dim(V\otimes W)=4.

v= e_1+2e_2, \qquad w= 3e_1-e_2,

then

v\otimes w = (e_1+2e_2)\otimes(3e_1-e_2).

Using bilinearity,

= 3e_1\otimes e_1 - e_1\otimes e_2 + 6e_2\otimes e_1 - 2e_2\otimes e_2.

This expansion illustrates how tensor products distribute across linear combinations.

98.18 Summary

Tensor products combine vector spaces into larger spaces that encode multilinear structure.

The tensor product:

Concept	Meaning
$V\otimes W$	Space generated by bilinear combinations
$v\otimes w$	Pure tensor
Universal property	Bilinear maps become linear
Basis tensors	Products of basis vectors
Kronecker product	Matrix realization of tensor products
Tensor rank	Minimal pure decomposition

Tensor products unify algebra, geometry, analysis, physics, and computation. They provide the language for multilinear structures and higher-dimensional interactions. Much of modern mathematics can be viewed as the systematic study of structures built from tensor products.