Page 291 - Statistics for Environmental Engineers

L1592_frame_C33 Page 297 Tuesday, December 18, 2001 2:51 PM

Residuals that are normally and independently distributed with constant variance over the range of values
studied are persuasive evidence that the proposed model adequately fits the data. If the residuals show
some pattern, the pattern will suggest how the model should be modified to improve the fit. One way to
check the adequacy of the model is to check the properties of the residuals of the fitted model by plotting
them against the predicted values and against the independent variables.

The Method of Least Squares
The best estimates of the model parameters are those that minimize the sum of the squared residuals:

n n
S = ∑ () = ∑ ( y i η i ) 2
2
–
e i
i=1 i=1
(
The minimum sum of squares is called the residual sum of squares S R ) . This approach to estimating
the parameters is known as the method of least squares. The method applies equally to linear and
nonlinear models. The difference between linear and nonlinear regression lies in how the least squares
parameter estimates are calculated. The essential difference is shown by example.
Each term in the summation is the difference between the observed y i and the η computed from the
()
model at the corresponding values of the independent variables x i . If the residuals e i are normally and
independently distributed with constant variance, the parameter estimates are unbiased and have mini-
mum variance.
For models that are linear in the parameters, there is a simple algebraic solution for the least squares
parameter estimates. Suppose that we wish to estimate β in the model η = βx . The sum of squares
function is:
S β() = ∑ ( y i βx i ) = ∑ ( y i – 2βx i y i β x i )
+
2 2
2
2
–
The parameter value that minimizes S is the least squares estimate of the true value of β. This estimate
is denoted by b. We can solve the sum of squares function for this estimate b() by setting the derivative
with respect to β equal to zero and solving for b:
dS β()
-------------- = 0 = 2 ∑ ( bx i – x i y i )
2
dβ
This equation is called the normal equation. Note that this equation is linear with respect to b. The
algebraic solution is:

b = ∑x i y i
-------------
2
∑x i
Because x i and y i are known once the experiment is complete, this equation provides a generalized method
for direct and exact calculation of the least squares parameter estimate. (Warning: This is not the equation
for estimating the slope in a two-parameter model.)
If the linear model has two (or more) parameters to be estimated, there will be two (or more) normal
equations. Each normal equation will be linear with respect to the parameters to be estimated and
therefore an algebraic solution is possible. As the number of parameters increases, an algebraic solution
is still possible, but it is tedious and the linear regression calculations are done using linear algebra (i.e.,
matrix operations). The matrix formulation was given in Chapter 30.
Unlike linear models, no unique algebraic solution of the normal equations exists for nonlinear models.
For example, if η = exp θx), the method of least squares requires that we find the value of θ that
(
–
minimizes S:
S θ() = ∑ ( y i – exp – θx i )) = ∑ [ y i – 2y i exp −θx i ) + ( exp θx i )) ]
– (
(
(
2
2
2
© 2002 By CRC Press LLC