Page 802 - Mechanical Engineers' Handbook (Volume 2)

2 Background 793

Dendrites

Axon terminals
Nucleus
Node of Ranvier
Myelin

Axon Synapses
Cell body

Figure 1 Nervous system cell. (With permission from http://www.sirinet.net/ jgjohnso/index.html.)

2 BACKGROUND
2.1 Neural Networks

The multilayer NN is modeled based on the structure of biological nervous systems (see
n
m
Fig. 1) and provides a nonlinear mapping from an input space R into an output space R .
Its properties include function approximation, learning, generalization, classification, and so
on. It is known that the two-layer NN has sufficient generality for closed-loop control pur-
poses. The two-layer NN shown in Fig. 2 consists of two layers of weights and thresholds
and has a hidden layer and an output layer. The input function x(t) has n components, the
hidden layer has L neurons, and the output layer has m neurons.
One may describe the NN mathematically as
T
T
y W (Vx)
where V is a matrix of first-layer weights and W is a matrix of second-layer weights. The
second-layer thresholds are included as the first column of the matrix W by augmenting
T
the vector activation function ( ) by 1 in the first position. Similarly, the first-layer thresh-
T
olds are included as the first column of the matrix V by augmenting vector x by 1 in the
first position.
The main property of NNs we are concerned with for control and estimation purposes
is the function approximation property. 2,3 Let ƒ(x) be a smooth function from R → R . Then,
m
n
it can be shown that if the activation functions are suitably selected and is restricted to a
compact set S R , then for some sufficiently large number L of hidden-layer neurons, there
n
exist weights and thresholds such that one has
T
T
ƒ(x) W (Vx) (x)
with (x) suitably small. Here, (x) is called the neural network functional approximation
error. In fact, for any choice of a positive number , one can find a NN of large enough
N
size L such that (x) for all x S.
N
Finding a suitable NN for approximation involves adjusting the parameters V and W to
obtain a good fit to ƒ(x). Note that tuning of the weights includes tuning of the thresholds
as well. The neural net is nonlinear in the parameters V, which makes adjustment of these
parameters difficult and was initially one of the major hurdles to be overcome in closed-