Page 177 - Artificial Intelligence in the Age of Neural Networks and Brain Computing

2. Brief History and Foundations of the Deep Learning Revolution 167

2.2 HOW THE DEEP REVOLUTION ACTUALLY HAPPENED
Why and how did the deep learning revolution occur when it did? It is amazing at
times to hear the varieties of creative stories which people swear by on these
topicsdbut certainly we know that Google’s big investment and energy were
the start of the big new wave, and Sergey Brin himself has told us how that
happened [3]. It was not a matter of strange politics or impulses, but of solid
breakthrough results which came to his attention. From the New York Times inves-
tigation of these events [10], we know that Andrew Ng came to Google with solid
proof that neural networks could solve problems in computer technology which
people had believed impossible before he came to them with proof. But how did
that happen?
In the year 2008, it was “generally known” that artificial neural networks could
not possibly perform well on tasks requiring pattern recognition over images with a
large number of pixels, or natural language processing, etc. It was also well-known
that CoNNs, bottleneck networks, and networks with many layers had been in exis-
tence for many decades (for example, see my 1988 paper in the flagship journal of
IEEE [14] citing early breakthrough success of CoNNs on small but challenging
tasks in ZIP code recognition, describing how to train a generalized network design
which effectively has N layers where N is the number of neurons). The new NSF
initiative depicted in Fig. 8.1 was intended to break down many of the cultural
barriers between disciplines which had limited progress in all these fields, so it
was only natural that Yan LeCun (like me, one of the early pioneers in using
backpropagation) expressed interest in submitting a proposal to obtain the funding
necessary to disprove the conventional wisdom and break the logjam. Because I
did have to follow the rules of NSF, I told him this would be fine, IF he could
find a card-carrying engineer to act as PI, with him as co-PI. The final proposal
from Ng and LeCun then came to the usual NSF panel review.
After the panel review, I was under incredible pressure not to fund that proposal.
After all, many more prominent people swore that this simply could not work, so we
should not waste money on bothering to find out. They could think of other uses for
the $2 million. Fortunately, Dr. Sohi Rastegar, head of the Emerging Frontiers
Research and Innovation(EFRI) office, stood by the great corporate culture which
had guided NSF ever since its creation by Vannevar Bush, and he did authorize
me to use my judgment and fund the proposal anyway, despite threats of lawsuits.
Once Ng and LeCun had enough funds to really attack big outstanding challenge
problems in computer technology, they reported breakthrough results defying all
conventional wisdom within just 1 year, as depicted in Fig. 8.4. This was just one
of the success reports from one of the four projects funded under the initiative
depicted in Fig. 8.1.
In summary, it was a solid demonstration of empirical results on widely used
competitions which really created this cultural revolution. Of course, the availability
of Game Processing Units (GPUs) also was important in making these demonstra-
tions possible. The open source culture of Google also was essential in having a