Page 14 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

CHAPTER

Nature’s Learning Rule:

The Hebbian-LMS 1

Algorithm

Bernard Widrow, Youngsik Kim, Dookun Park, Jose Krause Perin
Department of Electrical Engineering, Stanford University, Stanford, CA, United States

CHAPTER OUTLINE
1. Introduction ......................................................................................................... 1
2. ADALINE and the LMS Algorithm, From the 1950s .................................................. 3
3. Unsupervised Learning With Adaline, From the 1960s............................................ 5
4. Robert Lucky’s Adaptive Equalization, From the 1960s ........................................... 7
5. Bootstrap Learning With a Sigmoidal Neuron .......................................................10
6. Bootstrap Learning With a More “Biologically Correct” Sigmoidal Neuron .............13
6.1 Training a Network of Hebbian-LMS Neurons......................................... 17
7. Other Clustering Algorithms ................................................................................20
7.1 K-Means Clustering ............................................................................. 20
7.2 Expectation-Maximization Algorithm ..................................................... 21
7.3 Density-Based Spatial Clustering of Application With Noise Algorithm ..... 21
7.4 Comparison Between Clustering Algorithms ........................................... 21
8. A General Hebbian-LMS Algorithm.......................................................................21
9. The Synapse ......................................................................................................22
10. Postulates of Synaptic Plasticity .........................................................................25
11. The Postulates and the Hebbian-LMS Algorithm ...................................................26
12. Nature’s Hebbian-LMS Algorithm.........................................................................26
13. Conclusion.........................................................................................................27
Appendix: Trainable Neural Network Incorporating Hebbian-LMS Learning ...................27
Acknowledgments.....................................................................................................29
References ...............................................................................................................29

1. INTRODUCTION
Donald O. Hebb has had considerable influence in the fields of psychology and
neurobiology since the publication of his book The Organization of Behavior in
1949 [1]. Hebbian learning is often described as: “neurons that fire together wire
together.” Now imagine a large network of interconnected neurons whose synaptic
weights are increased because the presynaptic neuron and the postsynaptic neuron

1
Artificial Intelligence in the Age of Neural Networks and Brain Computing. https://doi.org/10.1016/B978-0-12-815480-9.00001-3
Copyright © 2019 Elsevier Inc. All rights reserved.