Chapter 51. Orthogonal Projections

An orthogonal projection is the operation of replacing a vector by its closest vector in a chosen subspace. It is the precise linear algebra version of dropping a perpendicular from a point to a line, plane, or higher-dimensional subspace.

If $W$ is a subspace of an inner product space $V$, then every vector $v$ can often be decomposed into two parts:

where

The vector $w$ is the orthogonal projection of $v$ onto $W$. The vector $r$ is the residual. In finite-dimensional inner product spaces, this decomposition exists and is unique for every subspace $W$. The projected vector is the closest vector in $W$ to the original vector.

51.1 Projection onto a Line

Let $u$ be a nonzero vector in an inner product space $V$. The line generated by $u$ is

The projection of $v$ onto $L$ is the vector in the direction of $u$ closest to $v$. It has the form

for some scalar $c$.

The residual is

For $cu$ to be the orthogonal projection, the residual must be orthogonal to the line. Since the line is spanned by $u$, it is enough to require

Using linearity,

Therefore

So the projection is

This formula is valid whenever $u\ne 0$.

51.2 Projection onto a Unit Vector

If $q$ is a unit vector, then

The projection formula simplifies to

This is the simplest projection formula. The scalar

is the coordinate of $v$ in the direction $q$. The vector

is the component of $v$ along $q$.

For example, let

Then

so

The projection keeps the horizontal component and removes the vertical component.

51.3 The Residual

The residual of $v$ after projection onto a subspace $W$ is

The defining property of orthogonal projection is

Equivalently,

for every $w\in W$.

Thus projection separates a vector into an explained part and an unexplained part:

where

This is the orthogonal decomposition of $v$ with respect to $W$.

51.4 Projection onto an Orthonormal Basis

Let $W$ be a subspace with orthonormal basis

The projection of $v$ onto $W$ is

This formula follows from the coordinate formula for orthonormal bases. The projected vector lies in $W$, and the residual is orthogonal to every $q_j$.

Indeed, let

Then for each $i$,

Using orthonormality,

Hence

Since the $q_i$ span $W$, the residual is orthogonal to all of $W$.

51.5 Projection Matrix for an Orthonormal Basis

Let $Q$ be the matrix whose columns are the orthonormal vectors

Then

The projection of $v$ onto $\operatorname{Col}(Q)$ is

Thus the projection matrix is

This formula is important because it expresses projection as matrix multiplication.

The matrix $P$ satisfies

Indeed,

It also satisfies

Therefore $P$ is symmetric and idempotent. A real matrix with these two properties is an orthogonal projection matrix.

51.6 Idempotence

A projection is idempotent. This means that applying it twice gives the same result as applying it once:

The reason is geometric. Once a vector has been projected onto a subspace, projecting it onto the same subspace again changes nothing.

If

and $p\in W$, then

Therefore

In matrix form,

for every vector $v$, so

Idempotence is the algebraic signature of projection. General projections are idempotent linear maps, while orthogonal projections also respect the inner product geometry.

51.7 Symmetry

A real projection matrix $P$ is an orthogonal projection matrix precisely when it is both idempotent and symmetric:

The idempotent condition says that $P$ is a projection. The symmetry condition says that the projection is orthogonal rather than oblique.

For complex matrices, symmetry is replaced by self-adjointness:

Here $P^*$ denotes the conjugate transpose.

Orthogonal projection matrices preserve the perpendicular relationship between the range and the residual. If $p=Pv$, then

This is the geometric content of the symmetry condition.

51.8 Projection onto a Column Space

Let $A$ be an $m\times n$ real matrix with linearly independent columns. We want the projection of $b\in\mathbb{R}^m$ onto

The projected vector has the form

for some $\hat{x}\in\mathbb{R}^n$.

The residual is

For $p$ to be the orthogonal projection, the residual must be orthogonal to every column of $A$. This condition is

Rearranging gives the normal equations:

Since the columns of $A$ are linearly independent, $A^TA$ is invertible. Thus

Therefore

The projection matrix onto $\operatorname{Col}(A)$ is

51.9 Why $A^TA$ Is Invertible

Assume the columns of $A$ are linearly independent. Then $A^TA$ is invertible.

To see this, suppose

Multiply on the left by $x^T$:

But

Hence

Therefore

Since the columns of $A$ are linearly independent, the null space of $A$ is trivial. Thus

So the null space of $A^TA$ is trivial, and $A^TA$ is invertible.

51.10 Projection Matrix onto a Column Space

For a full-column-rank matrix $A$, the projection matrix

has two key properties.

First,

Indeed,

Since

we get

Second,

This follows because $A^TA$ is symmetric, so its inverse is symmetric:

Thus $P$ is an orthogonal projection matrix.

51.11 Closest Vector Property

Orthogonal projection gives the closest vector in a subspace.

Let $W$ be a finite-dimensional subspace of an inner product space $V$. Let

Then

For any other vector $w\in W$,

Here

and

Therefore the two vectors $v-p$ and $p-w$ are orthogonal. By the Pythagorean theorem,

Since

we have

Thus

So $p$ is the closest vector in $W$ to $v$. Equality occurs only when $w=p$. This is the best approximation property of orthogonal projection.

51.12 Distance to a Subspace

The distance from $v$ to a subspace $W$ is

When $W$ is finite-dimensional, this infimum is attained by the orthogonal projection:

If $p=\operatorname{proj}_W(v)$, then

The residual is therefore the shortest error vector. It measures exactly how far $v$ lies from the subspace.

51.13 Least Squares

Orthogonal projection is the geometric core of least squares.

Consider an inconsistent system

If $b\notin \operatorname{Col}(A)$, there is no exact solution. Instead, we seek $\hat{x}$ such that

is as close as possible to $b$. This means minimizing

The closest vector in $\operatorname{Col}(A)$ is the orthogonal projection of $b$ onto $\operatorname{Col}(A)$. Therefore

The residual

must be orthogonal to $\operatorname{Col}(A)$. Hence

Substituting $r=b-A\hat{x}$ gives

or

These are the normal equations.

51.14 Example: Projection onto a Line in $\mathbb{R}^2$

Let

The projection of $v$ onto $\operatorname{span}\{u\}$ is

Compute

and

Thus

The residual is

Check orthogonality:

Thus the decomposition is

with the first vector on the line and the second vector perpendicular to the line.

51.15 Example: Projection onto a Plane

Let $W\subseteq \mathbb{R}^3$ be the $xy$-plane:

For

the projection onto $W$ is

The residual is

The projection matrix is

Then

This matrix satisfies

So it is an orthogonal projection matrix.

51.16 Example: Projection Using a Matrix

Let

The column space of $A$ is the line in $\mathbb{R}^3$ spanned by

The projection matrix is

Compute

Thus

For

the projection is

The residual is

Check orthogonality to the column of $A$:

Thus $p$ is the orthogonal projection of $b$ onto $\operatorname{Col}(A)$.

51.17 Orthogonal Projection and Coordinates

If $W$ has an orthonormal basis $q_1,\ldots,q_k$, then the projection coefficients are

Thus

These coefficients are the coordinates of the projected vector in the orthonormal basis of $W$.

The projection discards all components of $v$ in $W^\perp$. If $V=W\oplus W^\perp$, and

then

Projection is therefore a coordinate-selection operation relative to an orthogonal decomposition.

51.18 Orthogonal Projection and Energy

Let $p=\operatorname{proj}_W(v)$ and $r=v-p$. Since

the Pythagorean theorem gives

Thus projection splits the squared norm into two parts:

If $W$ has orthonormal basis $q_1,\ldots,q_k$, then

Therefore

This form appears in approximation theory, signal processing, Fourier analysis, statistics, and numerical linear algebra.

51.19 Oblique Projections

A projection need not be orthogonal.

A linear map $P:V\to V$ is a projection if

This only means that applying $P$ twice is the same as applying it once. It does not require the residual to be orthogonal to the range.

An oblique projection is a projection whose range and null space are complementary but not orthogonal.

For example,

satisfies

so it is a projection. But

so it is not an orthogonal projection.

Orthogonal projections are usually preferred when distance minimization matters. Oblique projections appear in other settings where the decomposition directions are prescribed by constraints rather than perpendicularity.

51.20 Projection Theorem

In finite-dimensional inner product spaces, every subspace $W$ has an orthogonal projection. For each $v\in V$, there is a unique vector $p\in W$ such that

This vector $p$ is the unique closest vector in $W$ to $v$.

In Hilbert spaces, the analogous result requires $W$ to be closed. If $M$ is a closed subspace of a Hilbert space $H$, then every $x\in H$ has a unique best approximation $\hat{x}\in M$, and the error $x-\hat{x}$ lies in $M^\perp$.

This result is called the projection theorem. It is the abstract form of the closest vector property.

51.21 Summary

Orthogonal projection decomposes a vector into a component inside a subspace and a residual perpendicular to that subspace:

For a line spanned by a nonzero vector $u$,

For a subspace with orthonormal basis $q_1,\ldots,q_k$,

For a full-column-rank matrix $A$, the projection matrix onto $\operatorname{Col}(A)$ is

Orthogonal projection gives the closest vector in a subspace, produces the residual used in least squares, and gives the geometric meaning of many matrix formulas.