Page 31 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

18 CHAPTER 1 Nature’s Learning Rule: The Hebbian-LMS Algorithm

If the input patterns were not linearly independent, the outputs of the first-layer
neurons would not be purely binary. The outputs of the second layer would be closer
to binary. The outputs of the third layer would be even closer to binary.
The number of training patterns equal to the number of synapses of the output
layer neurons is the capacity of the network. It is shown in Ref. [12] that when
applying patterns that are distinct but not necessarily linear independent to a
nonlinear process such as layers of a neural network, the outputs of the layers would
be distinct and linearly independent, if the number of training patterns were less than
or equal to the network capacity. Then the outputs of the output layer neurons will be
perfectly binary. If the number of training patterns exceeds the network capacity, the
network output will be 0s and “fuzzy” 1s, close to binary.
If one were to take one of the trained-in vectors and place a large cloud of vectors
randomly disposed in a cluster about it, inputting all the vectors in the cluster
without further training would result in identical binary output vectors at the output
layer. This will be true as long as the diameter of the cluster is not “too large.” How
large this could be depends on the number and disposition of the other training
vectors. Noisy or distorted input patterns in a cluster could be identified as equiva-
lent to the associated training pattern. The network could determine a unique output
pattern, a unique binary representation for training patterns in a cluster. This is a
useful property for pattern classification.
With unsupervised learning, each cluster “chooses” its own binary output repre-
sentation. The number of clusters that the network can resolve is equal to the
network capacity, equal to the number of weights of each of the neurons of the
output layer. Hebbian-LMS is a clustering algorithm.
An application is the following. Given a network trained by Hebbian-LMS, let
the weights be fixed. Inputting a pattern from one of the clusters of one of the
training patterns will result in a binary output vector. The sum of the squares of
the errors of the output neurons will be close to zero. Now, inputting a pattern not
close to any of the training patterns will result in an output vector that will not be
binary and the sum of the squares of the errors of the output neurons will be large.
So, if the input pattern is close to a training pattern, the output error will be close to
zero. If the input pattern is distinct from all the training patterns, the output error will
be large. One could use this when one is not asking the neural network to classify an
input pattern, merely to indicate if the input pattern has been trained in, that is, seen
before or not, deja vu, yes or no? This could be used as a critical element of a cogni-
tive memory system [13].
In another application, a multilayer neural network could be trained using both
supervised and unsupervised methods. The hidden layers could be trained with
Hebbian-LMS and the output layer could be trained with the original LMS algo-
rithm. An individual input cluster would produce an individual binary “word” at
the output of the final hidden layer. The output layer could be trained with a one-
out-of-many code. The output neuron with the largest (SUM) would be identified
as representing the class of the cluster of the input pattern.