Page 185 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

3. From RNNs to Mouse-Level Computational Intelligence: Next Big Things 175

the kind of recurrence one would use to represent models like simultaneous equation
models in economics, or implicit models defined by some kind of relaxation to equi-
librium. Feedforward CoNNs can approximate a wide range of smooth functions, but
for effective intelligent control and true brain-like intelligence it is important to have
a more general capability to approximate nonsmooth functions. Our 2008 paper gave
an example of a simple learning problem which arises in learning how to navigate a
cluttered space, as a general skill (not just learning a particular maze). The usual
CoNN simply could not learn such a difficult task, but the CSRN could. The
paper also described a further extension beyond CSRN suitable for exploiting
non-Euclidean symmetries, which is essential for true brain-like capabilities.
For an even more powerful universal learning system able to capture all the ben-
efits of the most advanced TLRN design and extensions of CSRN, one would simply
combine both types of recurrence in a single system (as does the brain). See the
Handbook of Intelligent Control [17] for discussion of how to combine both types
together.

3.2 DEEP VERSUS BROAD: A FEW PRACTICAL ISSUES
Philip Chen, a recent president of the IEEE Society for Systems, Man and Cyber-
netics, has argued this year [28] that “broad” neural networks could perform just
as well as “deep” (many layered) neural networks. In actuality, debates about the
number of layers were already resolved in principle by 1990, when I explained
[14] how to use the more general static ANN design illustrated in Fig. 8.11:
Formally, this is the deepest possible feedforward network possible with N
neurons and implementation on a sequential computer. Each neuron is a layer
unto itself, making it all broad as well as deep. All possible feedforward network
structures with N neurons are a special case of this general structure. The key is
to use rational, statistically grounded learning (using penalty functions, pruning
and growing methods) to learn the exact details. Such general methods have been

FIGURE 8.11
Generalized MLP design.