Page 300 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

CHAPTER

Evolving Deep

Neural Networks 15

Risto Miikkulainen 1,2 , Jason Liang 1,2 , Elliot Meyerson 1,2 , Aditya Rawal 1,2 ,
1
1
1
1
1
Daniel Fink , Olivier Francon , Bala Raju , Hormoz Shahrzad , Arshak Navruzyan ,
1
Nigel Duffy , Babak Hodjat 1
1
Sentient Technologies, Inc., San Francisco, CA, United States ; The University of Texas at Austin,
Austin, TX, United States 2
CHAPTER OUTLINE
1. Introduction .......................................................................................................293
2. Background and Related Work ............................................................................294
3. Evolution of Deep Learning Architectures.............................................................296
3.1 Extending NEAT to Deep Networks ....................................................... 296
3.2 Cooperative Coevolution of Modules and Blueprints................................ 297
3.3 Evolving DNNs in the CIFAR-10 Benchmark.......................................... 298
4. Evolution of LSTM Architectures..........................................................................300
4.1 Extending CoDeepNEAT to LSTMs........................................................ 300
4.2 Evolving DNNs in the Language Modeling Benchmark ............................ 301
5. Application Case Study: Image Captioning for the Blind........................................302
5.1 Evolving DNNs for Image Captioning..................................................... 303
5.2 Building the Application ...................................................................... 304
5.3 Image Captioning Results .................................................................... 305
6. Discussion and Future Work................................................................................307
7. Conclusion ........................................................................................................309
References .............................................................................................................309

1. INTRODUCTION

Large databases (i.e., Big Data) and large amounts of computing power have become
readily available since the 2000s. As a result, it has become possible to scale up
machine learning systems. Interestingly, not only have these systems been successful
in such scaleup, but they have become more powerful. Some ideas that did not quite
work before, now do, with million times more compute and data. For instance, deep
learning neural networks (DNNs), that is, convolutional neural networks (CNN) [1]
and recurrent neural networks (in particular, Long Short Term Memory, or LSTM
[2]), which have existed since the 1990s, have improved state of the art significantly
in computer vision, speech, language processing, and many other areas [3e5].
293
Artificial Intelligence in the Age of Neural Networks and Brain Computing. https://doi.org/10.1016/B978-0-12-815480-9.00015-3
Copyright © 2019 Elsevier Inc. All rights reserved.