Page 54 - Foundations of Cognitive Psychology : Core Readings

52 Daniel C. Dennett

appreciate that these models cast light on the deep theoretical question of how
the mind is organized.

Postscript [1997]

In 1991, the First Annual Loebner Prize Competition was held in Boston at the
Computer Museum. Hugh Loebner, a New York manufacturer, had put up the
money for a prize—a bronze medal and $100,000—for the first computer pro-
gram to pass the Turing test fair and square. The Prize Committee, of which I
was Chairman until my resignation after the third competition, recognized that
no program on the horizon could come close to passing the unrestricted test—
the only test that is of any theoretical interest at all, as this essay has explained.
So to make the competition interesting during the early years, some restrictions
were adopted (and the award for winning the restricted test was dropped to
$2,000). The first year there were ten terminals, with ten judges shuffling from
terminal to terminal, each spending fifteen minutes in conversation with each
terminal. Six of the ten contestants were programs, four were human ‘‘con-
federates’’ behind the scenes.
Each judge had to rank order all ten terminals from most human to least hu-
man. The winner of the restricted test would be the computer with the highest
mean rating. The winning program would not have to fool any of the judges,
nor would fooling a judge be in itself grounds for winning; highest mean
ranking was all. But just in case some program did fool a judge, we thought this
fact should be revealed, so judges were required to draw a line somewhere
across their rank ordering, separating the humans from the machines.
We on the Prize Committee knew the low quality of the contesting programs
that first year, and it seemed obvious to us that no program would be so lucky
as to fool a single judge, but on the day of the competition, I got nervous. Just
to be safe, I thought, we should have some certificate prepared to award to any
programmer who happened to pull off this unlikely feat. While the press and
the audience were assembling for the beginning of the competition, I rushed
into a back room at the Computer Museum with a member of the staff and we
cobbled up a handsome certificate with the aid of a handy desktop publisher.
In the event, we had to hand out three of these certificates, for a total of seven
positive misjudgments out of a possible sixty! The gullibility of the judges was
simply astonishing to me. How could they have misjudged so badly? Here I had
committed the sin I’d so often found in others:treating a failure of imagination
as an insight into necessity. But remember that in order to make the competi-
tion much easier, we had tied the judges’ hands in various ways—too many
ways. The judges had been forbidden to probe the contestants aggressively, to
conduct conversational experiments. (I may have chaired the committee, but I
didn’t always succeed in persuading a majority to adopt the rules I favored.)
When the judges sat back passively, as instructed, and let the contestants lead
them, they were readily taken in by the Potemkin village effect described in the
essay.
None of the misjudgments counted as a real case of a computer passing
the unrestricted Turing test, but they were still surprising to me. In the second
year of the competition, we uncovered another unanticipated loophole:due to