Page 203 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

2. Emergence of Some Neural Network Principles 193

strength if stimuli were unpaired [85]. In fact, the findings on long-term potentiation
at synapses were shortly followed by other findings on long-term depression
[16e18].
A synthesis of Hebb and McCulloch-Pitts occurred in various versions of the
perceptrons due to Rosenblatt [19] whose multilayer perceptrons were trained to
respond selectively to specific patterns of sensory stimuli. Perceptrons formed the
basis for the widely used back propagation networks [4,20], which in turn form
the basis for the currently popular deep learning methods [21e23].
The mathematics employed in the McCulloch-Pitts and Rosenblatt models was
discrete, as befits a network of all-or-none neurons. Yet the biophysics of neurons,
like other physical processes, points to the importance of variables that can take
on a continuum of possible values. Rashevsky [24] and other pioneers showed
that the discrete processes in a network of neurons could be averaged over a large
number of neurons to generate continuous processes in networks whose units are
populations rather than individual neurons.

2. EMERGENCE OF SOME NEURAL NETWORK PRINCIPLES
From about the mid-1960s to the mid-1980s, when much less-detailed cognitive
neuroscience results were available than are now, several pioneering neural
modelers developed network architectures based largely on abstract principles,
through incorporating some of that period’s state of the art in neuroscience
(cf. Ref. [1], Chapters 3 and 4). This led to the development of some “building
blocks” that could be incorporated into larger and more complex networks. Since
different building blocks were used to model different functions, this was not a
“one size fits all” approach to neural modeling.
One of these principles, inspired both by Hebb and by psychologists such as Hull
[13], was associative learning. Associative learning was a prominent feature in the
work of Grossberg, Anderson, Kohonen, and Kosko [25e38]. Grossberg’s early
networks, such as the outstar, were particularly notable for including conditioned
decreases as well as increases in synaptic strength, in order that a network could learn
the proportions between activations of different units in a stimulus pattern.
Just as associative learning is important in models of long-term memory, lateral
inhibition or competition plays a crucial role in models of short-term memory. This
principle is particularly important in models of perception [39e43].Competitive
interactions capture data showing that the visual system does not store every stimulus
it receives equally but sharpens contrast and reduces noise.
Opponent processing is a psychological principle that captures the fact that people
and animals often perceive contrasts rather than absolute amounts of variables.
Contrast effects were first observed in the realm of emotion: absence of electric shock
when one has just been shocked is rewarding, and absence of food when one is
hungry and expecting food is punishing. Grossberg ([5]), explained these effects
using the gated dipole model with neural transmitter accumulation and depletion,