Page 303 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

296 CHAPTER 15 Evolving Deep Neural Networks

better performance. In this manner, evolution was harnessed in this example task to
optimize a system design that was too complex to be optimized by hand.
Very recently, studies have started to emerge with the goal of optimizing DNNs.
Due to limited computational resources, they have focused on specific parts of the
design. For instance, Loshchilov and Hutter [29] used CMA-ES to optimize the
hyperparameters of existing DNNs, obtaining state-of-the-art results on, for
example, object recognition. Further, Fernando et al. [30] evolved a CPPN (compo-
sitional pattern-producing network [31]) to output the weights of an autoencoder
neural network. The autoencoder was then trained further through gradient descent,
forming gradients for the CPPN training, and its trained weights were then incorpo-
rated back into the CPPN genome through Lamarckian adaptation. A related
approach was proposed by Zoph and Le [32]: the topology and hyperparameters
of a deep network and LSTM network were modified through policy iteration.
Building on this foundation, a systematic approach to evolving DNNs is devel-
oped in this paper. First, the standard NEAT neuroevolution method is applied to the
topology and hyperparameters of CNNs, and then extended to evolution of compo-
nents as well, achieving results comparable to state of the art in the CIFAR-10 image
classification benchmark. Second, a similar method is used to evolve the structure of
LSTM networks in language modeling, showing that even small innovations in the
components can have a significant effect on performance. Third, the approach is
used to build a real-world application on captioning images in the website for an on-
line magazine.

3. EVOLUTION OF DEEP LEARNING ARCHITECTURES

NEAT neuroevolution method [12] is first extended to evolving network topology
and hyperparameters of deep neural networks in DeepNEAT, and then further to
coevolution of modules and blueprints for combining them in CoDeepNEAT. The
approach is tested in the standard CIFAR-10 benchmark of object recognition,
and found to be comparable to the state of the art.

3.1 EXTENDING NEAT TO DEEP NETWORKS
DeepNEAT is a most immediate extension of the standard neural network
topologyeevolution method NEAT to DNN. It follows the same fundamental pro-
cess as NEAT: first, a population of chromosomes (each represented by a graph)
with minimal complexity is created. Over generations, structure (i.e., nodes and
edges) is added to the graph incrementally through mutation. During crossover, his-
torical markings are used to determine how genes of two chromosomes can be lined
up. The population is divided into species (i.e., subpopulations) based on a similarity
metric. Each species grows proportionally to its fitness and evolution occurs sepa-
rately in each species.