Orthogonal Projections and Orthogonal Bases

Orthogonal sets

A set of vectors is orthogonal when every pair has dot product $0$. The standard basis of ℝⁿ is one example; so is

$$\mathbf{u}_1 = (3, 1, 1) \qquad \mathbf{u}_2 = (-1, 2, 1) \qquad \mathbf{u}_3 = \left(-\tfrac{1}{2}, -2, \tfrac{7}{2}\right)$$

since $\mathbf{u}_1 \cdot \mathbf{u}_2 = -3 + 2 + 1 = 0$, and the other two pairs also give $0$.

An orthogonal set of nonzero vectors is automatically linearly independent. If $c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p = \mathbf{0}$, taking the dot product of both sides with $\mathbf{u}_j$ kills every term but one, leaving $c_j\,\mathbf{u}_j \cdot \mathbf{u}_j = 0$, so $c_j = 0$.

The same move finds coordinates. If $\mathbf{y} = c_1\mathbf{u}_1 + \cdots + c_p\mathbf{u}_p$, dotting with $\mathbf{u}_j$ isolates $c_j$:

$$c_j = \frac{\mathbf{y} \cdot \mathbf{u}_j}{\mathbf{u}_j \cdot \mathbf{u}_j}$$

Projecting onto a line

The dot product lesson found the shadow of one vector on another. For a line $L$ spanned by $\mathbf{u}$,

$$\hat{\mathbf{y}} = \text{proj}_L\,\mathbf{y} = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\,\mathbf{u}$$

and the leftover $\mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to $\mathbf{u}$. For $\mathbf{y} = (7, 6)$ and $\mathbf{u} = (4, 2)$ the ratio is $\tfrac{28 + 12}{16 + 4} = 2$, so $\hat{\mathbf{y}} = (8, 4)$, and the leftover $(-1, 2)$ has dot product $-4 + 4 = 0$ with $\mathbf{u}$. The distance from $\mathbf{y}$ to the line is $\|(-1, 2)\| = \sqrt{5}$.

• line spanned by (4, 2)

Why the projection is the closest point

For a subspace $W$ with an orthogonal basis $\mathbf{u}_1, \ldots, \mathbf{u}_p$, the projection adds up the projections onto each basis vector:

$$\hat{\mathbf{y}} = \frac{\mathbf{y} \cdot \mathbf{u}_1}{\mathbf{u}_1 \cdot \mathbf{u}_1}\,\mathbf{u}_1 + \cdots + \frac{\mathbf{y} \cdot \mathbf{u}_p}{\mathbf{u}_p \cdot \mathbf{u}_p}\,\mathbf{u}_p$$

Dotting $\mathbf{y} - \hat{\mathbf{y}}$ with any $\mathbf{u}_j$ gives $\mathbf{y} \cdot \mathbf{u}_j - c_j\,\mathbf{u}_j \cdot \mathbf{u}_j = 0$, so the error is orthogonal to every basis vector and therefore to all of $W$. Now take any other point $\mathbf{w}$ in $W$. The vector $\mathbf{y} - \mathbf{w}$ splits as $(\mathbf{y} - \hat{\mathbf{y}}) + (\hat{\mathbf{y}} - \mathbf{w})$, the first piece orthogonal to $W$ and the second inside it, so by the Pythagorean theorem

$$\|\mathbf{y} - \mathbf{w}\|^2 = \|\mathbf{y} - \hat{\mathbf{y}}\|^2 + \|\hat{\mathbf{y}} - \mathbf{w}\|^2 \ge \|\mathbf{y} - \hat{\mathbf{y}}\|^2$$

Because the error $\mathbf{y} - \hat{\mathbf{y}}$ is perpendicular to $W$, every other point of $W$ is farther from $\mathbf{y}$: the projection is the best approximation to $\mathbf{y}$ from inside $W$.

Worked examples

Common mistakes

Practice problems