Page 443 - Matrix Analysis & Applied Linear Algebra

5.13 Orthogonal Projection 439

Least Squares Solutions

Each of the following four statements is equivalent to saying that : x is a
least squares solution for a possibly inconsistent linear system Ax = b.
• A: x − b = min Ax − b . (5.13.16)
2 2
x∈ n
• A: x = P R(A) b. (5.13.17)
T
T
∗
∗
• A A: x = A b ( A A: x = A b when A ∈C m×n ). (5.13.18)
†
†
• : x ∈ A b + N (A)( A b is the minimal 2-norm LSS). (5.13.19)
Caution! These are valuable theoretical characterizations, but none is
recommended for floating-point computation. Directly solving (5.13.17)
†
or (5.13.18) or explicitly computing A can be inefficient and numeri-
cally unstable. Computational issues are discussed in Example 4.5.1 on
p. 214; Example 5.5.3 on p. 313; and Example 5.7.3 on p. 346.
The least squares story will not be complete until the following fundamental
question is answered: “Why is the method of least squares the best way to make
estimates of physical phenomena in the face of uncertainty?” This is the focal
point of the next section.
Exercises for section 5.13

5.13.1. Find the orthogonal projection of b onto M = span {u} , and then de-
T
termine the orthogonal projection of b onto M , where b =( 4 8 )
⊥
T
and u =( 3 1 ) .
   
120 1
1
5.13.2. Let A =  241  and b =   .
120 1
(a) Compute the orthogonal projectors onto each of the four funda-
mental subspaces associated with A.
(b) Find the point in N (A) ⊥ that is closest to b.

5.13.3. Foran orthogonal projector P, prove that Px = x if and only
2 2
if x ∈ R (P).

T
5.13.4. Explain why A P R(A) = A T for all A ∈ m×n .