Page 80 - Compact Numerical Methods For Computers

Some comments on the formation of the cross-products matrix 69
In six-digit rounded computation this produces

which is singular since the first two columns or rows are identical. If we use
deviations from means (and drop the constant column) a singular matrix still
results. For instance, on a Data General NOVA minicomputer using a 23-bit
binary mantissa (between six and seven decimal digits), the A matrix using
deviation from mean data printed by the machine is

and the cross-products matrix as printed is

which is singular.
However, by means of the singular-value decomposition given by algorithm 1,
the same machine computes the singular values of A (not A') as
2·17533, 1·12603 and 1E–5.

Since the ratio of the smallest to the largest of the singular values is only slightly
larger than the machine precision (2 -22 2·38419E-7), it is reasonable to pre-
sume that the tolerance q in the equation (3.37) should be set to some value
between 1E-5 and 1·12603. This leads to a computed least-squares solution

with a residual sum of squares
T
r r = 1·68955E–5.
With the tolerance of q = 0, the computed solution is

with
T
r r = 1·68956E–4.
(In exact arithmetic it is not possible for the sum of squares with q= 0 to exceed
that for a larger tolerance.)