Page 39 - Rapid Learning in Robotics

3.1 A Brief History and Overview of Neural Networks 25

In (1969) Minsky and Papert showed that certain classes of problems,
e.g. the “exclusive-or” problem, cannot be learned with the simple percep-
tron. They doubted that learning rules could be found for computation-
ally more powerful multi-layered networks and recommended to focus on
the symbolic oriented learning paradigm, today called artificial intelligence
(“AI”). The research funding for artificial neural networks was cut, and it
took twenty years until the field became viable again.
An important stimulus for the field was the multiple discovery of the
error back-propagation algorithm. Its has been independently invented
in several places, enabling iterative learning for multi-layer perceptrons
(Werbos 1974, Rumelhart, Hinton, and Williams 1986, Parker 1985). The
MLP turned out to be a universal approximator, which means that using
a sufficient number of hidden units, any function can be approximated
arbitrarily well. In general two hidden layers are required - for continuous
functions one layer is sufficient (Cybenko 1989, Hornik et al. 1989). This
property is of high theoretical value, but does not guarantee efficiency of
any kind.
Other important developments where made: e.g. v.d. Malsburg and
Willshaw (1977, 1973) modeled the ordered formation of connections be-
tween neuron layers in the brain. A strongly related, more formal algo-
rithm was formulated by Kohonen for the development of a topographi-
cally ordered map from a general space of input stimuli to a layer of ab-
stract neurons. We return to Kohonen's work later in Sec. 3.7.
Hopfield (1982, 1984) contributed a famous model of the content-addressable
Hopfield network, which can be used e.g. as associative memory for im-
age completion. By introducing an energy function, he opened the mathe-
matical toolbox of statistical mechanics to the class of recurrent neural net-
works (mean field theory developed for the physics of magnetism). The
Boltzmann machine can be seen as a generalization of the Hopfield net-
work with stochastic neurons and symmetric connection between the neu-
rons (partly visible – input and output units – and partly hidden units).
“Stochastic” means that the input influences the probability of the two
possible output states (y g) which the
f neuron can take (spin glass
like).
The Radial Basis Function Networks (“RBF”) became popular in the
connectionist community by Moody and Darken (1988). The RFB belong
to the class of local approximation schemes (see p. 33). Similarities and