The Uncanny Valley and the ‘Flaw’ in the Turing Test

“A Great science fiction detective story” – Ian Watson, author of The Universal Machine

NOW AVAILABLE for instant download! Click to find out more.

Days to Centenary: 3

3… 2… 1…!
Just 3 short days to the Turing Centenary!

Criminy! We are a few short paces away from the big day — if I squint, I can just see it coming over the horizon.

And even as we count down the last few days, new events, articles, and blog posts come in.

The Flaw in the Turing Test

Just yesterday Terry Walby, the UK managing director at IPsoft, an IT service company, had a guest post on the Wired Science blog entitled Why the Turing Test Is a Flawed Benchmark.

The main thrust of Walby’s argument seems to be that Turing was misguided in recommending that we measure the ability of a machine to think by using human intelligence as a standard:

But Turing was wrong. A machine should not demonstrate intelligence by emulating a human. In fact, in some regards today’s expert systems are displaying intelligence far beyond the capability of a human. Should we mask such intellectual prowess in order for the machine to appear human, or allow it to run free to reach its full potential?

So is the Turing Test flawed and — as Walby later suggests — in need of replacing with a more satisfactory process?

How the Turing Test Works

First, for those who are new to this area — many people are learning about Turing and his work for the first time this year — a quick review of how the Turing Test works.

The test, or the Imitation Game as Turing himself called it, requires three participants:

1. a human judge
2. a hidden human who communicates with the judge only in writing, basically by text message
3. a hidden artificial intelligence that similarly communicates with the judge only in writing

The judge knows that either participant 2 or participant 3 is a computer while the other is human, and 2 and 3 both have to try to convince the judge that they’re the human being.  If the computer succeeds — if it can act human enough to fool a human judge — it has passed the Turing Test and has earned the right to be treated as intelligent without any consideration of the means by which it managed that persuasion.

Turing introduces the idea of the Imitation Game to the reader gradually by first having the hidden participants be a man and a woman, with the judge having to figure out which is which.  This is a parlour game version of the Imitation Game.

He then replaces the woman with a machine to turn the parlour game into a scientific enquiry and get at the question of machine intelligence. Remember that the paper was published in 1950 when Turing was in the process of inventing the discipline of artificial intelligence, so at the time this process would have eased readers into unfamiliar territory.

A copy of the issue of Mind in which Turing´s paper appeared (click to go to PDF)

Is the Flaw in the Turing Test Real?

So is Walby right?  This would be a boring post if I simply agreed with him, and overall I won’t (though his post is interesting and my critique is intended to be friendly and respectful).  But I want to start by agreeing on this point: machine intelligence should not be judged solely in comparison to human intelligence.

(One of my other blogs, Homo Artificialis, looks at disciplines that could eventually contribute to the creation of synthetic human bodies, artificial intelligence, or both, including AI, robotics, nanomedicine, brain-computer interfaces, artificial organs, tissue engineering, and xenotransplantation.  If you’ve visited Homo Artificialis you’ll know that I’m at least notionally sympathetic to the idea of free range artificial intelligence developing on its own terms into its own most realized form.)

Homo Artificialis Site

The trouble with Walby’s argument is that I don’t think Turing ever said that artificial intelligence should be judged by human standards — he simply never made the claim that Walby is disputing.

In his seminal paper  “Computing Machinery and Intelligence” [pdf] — the paper in which he codifies his famous test — Turing directly addressed the possibility that machines might ultimately be possessed of some form of intelligence unique to them and distinguishable from that of human beings:

May not machines carry out something which ought to be described as thinking but which is very different from what a man does?

He then simply puts this issue to one side, not because he’s dismissing it — he explicitly doesn’t dismiss it — but because it’s not the topic he’s addressing:

This objection is a very strong one, but at least we can say that if, nevertheless, a machine can be constructed to play the imitation game satisfactorily, we need not be troubled by this objection.

In other words, Turing agrees that machine intelligence may comprise different types, including some that do resemble human intelligence and some that don’t. The fact that there may be types that don’t simply doesn’t affect the subject of his enquiry: the types that do.

Indeed, while Turing famously starts the paper by asking “can machines think?”, later he is at pains to carefully circumscribe the question he’s addressing and to distinguish it from that larger, initial question:

We now ask the question, “What will happen when a machine takes the part of A [participant 3, above] in this game?”

Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman?

These questions replace our original, “Can machines think?” [emphasis added]

What the Turing Test Does

The Turing Test is not an exhaustive test for any and all kinds of artificial intelligence and I think it’s apparent that it wasn’t constructed to be.

What it is, is a test for a particular kind of evidence of artificial intelligence and it was carefully created to find the kind of evidence that is most persuasive to even the most skeptical of doubters.

We human beings ascribe intelligence to each other all the time even though we have no direct experience of another person’s intellect in action (a fact that Turing explicitly acknowledges in his discussion of the Argument from Consciousness).

We witness other people’s actions and hear or read their words, but that’s not conclusive of their engaging in thought. Maybe they’re actually hallucinations without intellects of their own, conjured up by our own minds. Or perhaps they’re illusions without substance projected by manipulative alien creatures in a Star Trek episode.

We have no direct evidence that other people think, but there is nonetheless a logic to our assumption that they do.  If you compare the actions and words of other people with your own, and find a high degree of similarity, it’s logical to conclude that since you have intelligence and they behave like you do, then they must have intelligence as well.

(We don’t actually think this process through, its an assumption we make, but making the assumption that other things that behave like you are like you is useful from the point of view of survival. Other animals do this as well, like a cat treating a wiggling piece of string as though it were living prey or hissing defensively at a self-propelled toy.)

This is a process in which we all engage and the strength of the Turing Test is that it takes this pre-existing reaction that we universally share and applies it to the question of machine intelligence.  It says: if and when a machine can do the things that we ourselves do, then at that point we will make the same assumption about the machine that we do about other people, that is, that it is thinking.

Seeing our own reflection in others

The Turing Test Doesn’t Need Turing to Function

When the Turing Test is viewed in this light, it can be seen not as Turing’s invention, but as his recognition of a naturally-occuring process that would eventually be applied to artificial constructs (once they were sophisticated enough to engage it) just as it’s always been applied to natural creatures.

Arguing with it makes little sense because it’s simply what we have always done and will continue to do: react to other things based upon their resemblance to us.

And by now our artificial constructs have finally become sophisticated enough to engage this instinct.  When we recognize the spooky near-humanity of some piece of  CGI that doesn’t quite fool us into thinking it’s a person, we’re giving it a failing grade in a kind of Turing Test that we automatically apply to the everything around us.

The tension and unease that arise when something almost passes the test, but doesn’t quite, was described in 1970 by Japanese robotics professor Masahiro Mori as the “uncanny valley,” [Wikipedia, Mori's paper] and it’s well illustrated by the video below.

.

Walby’s Argument for a New Turing Test

Terry Walby concludes that a new Turing Test is needed.  Given the arguments above, should we reject this conclusion?  I don’t think so.

If, as I’ve argued, Walby mistakes the Turing Test for something it isn’t, that doesn’t change the fact that the thing he’s calling for would be a damned useful thing to have.

Turing purposely sidestepped an exhaustive definition of “thinking” in order to get to a practical test for a particular kind of thinking — the kind humans do.

But thinking is not a unitary thing.  At a minimum, each of us experiences different kinds of thinking at different moments in our lives.  ”Thought” is not a point on a graph, it’s a blob that stretches along the X and Y axes (and possibly the Z as well), encompassing a variety of intellectual functions.

Any tool that helps us to explore, describe, and understand the territory that “thinking” maps on that graph is beneficial and worth working toward.

Wired Says a Turing Test Contender May Be on the Horizon

“A Great science fiction detective story”
- Ian Watson, author of The Universal Machine

Days to Centenary: 70

Just in time for the Turing Centenary and the Alan Turing Year, Wired magazine reported just days ago that we may soon see a genuine contender for an artificial intelligence that can pass the Turing Test, or as Turing himself called it, the Imitation Game.

If you´re new to this area — and many people are learning about Turing for the first time this year — the Turing Test in a nutshell requires three participants:

1. a human judge
2. a hidden human who communicates with the judge only in writing, basically by text message
3. a hidden artificial intelligence that similarly communicates with the judge only in writing

The judge knows that either participant 2 or participant 3 is a computer, while 2 and 3 both have to try to convince the judge that they´re the human being.  If the computer succeeds, it has passed the Turing Test and has earned the right to be treated as intelligent without any consideration of the means by which it managed that persuasion.

An image of the front of Turing´s groundbreaking paper

The test was set out in a paper entitled Computing Machinery and Intelligence, published in 1950, which effectively founded the discipline of artificial intelligence.

One rationale for the test is that it reproduces exactly the way in which we humans deal with one another.  Experientially, the only person I know is intelligent — whatever flaws there might be in that faculty — is me, because I know my own thoughts directly.  When I meet you, or anyone else, I can only judge what your internal life might be by interpreting your outward behaviour.

In other words, I have no way of knowing to a certainty that you´re a sentient being, but if you behave like one then I will tend to adopt the operational assumption that you are indeed one.

Why — Turing asked — should a machine be dealt with any differently?

A copy of the issue of Mind in which Turing´s paper appeared

For the most part the Turing Test has remained a hypothetical construct.  That is until a guy named Hugh Loebner established the Loebner Prize, in which actual attempts are made by computers to pass the test (for details see my post Turing Media Feast, Part III: The Turing Test in Theory and in Real Life).

A little friendly competition

That competition has not so far produced anything that is likely to pass the Turing Test any time soon.  A recent article in the esteemed-but-paywall-protected journal Science (Dusting Off the Turing Test, by Robert M. French. Science, Vol. 336 No. 6088, April 13, 2012) has postulated that that situation may soon change.

Given the paywall, I am relying on a report from Wired about the article:

“Two revolutionary advances in information technology may bring the Turing test out of retirement,” wrote Robert French, a cognitive scientist at the French National Center for Scientific Research, in an Apr. 12 Science essay. “The first is the ready availability of vast amounts of raw data — from video feeds to complete sound environments, and from casual conversations to technical documents on every conceivable subject. The second is the advent of sophisticated techniques for collecting, organizing, and processing this rich collection of data.”

Notoriously, the human mind proved to be less like a computer than had been thought in the mid-20th century, so success at the Turing Test was more problematic than expected, to the extent that many people dismissed the quest entirely, but the Science article appears to take the position that this was a mistake.  Says Wired:

Suppose, for a moment, that all the words you have ever spoken, heard, written, or read, as well as all the visual scenes and all the sounds you have ever experienced, were recorded and accessible, along with similar data for hundreds of thousands, even millions, of other people. Ultimately, tactile, and olfactory sensors could also be added to complete this record of sensory experience over time,” wrote French in Science, with a nod to MIT researcher Deb Roy’s recordings of 200,000 hours of his infant son’s waking development.

He continued, “Assume also that the software exists to catalog, analyze, correlate, and cross-link everything in this sea of data. These data and the capacity to analyze them appropriately could allow a machine to answer heretofore computer-unanswerable questions” and even pass a Turing test.

It´s a timely moment to revive Turing´s specualtions, given his approaching centenary.  It seems unlikely that the Turing Test will actually be passed this year, but 2012 would be a welcome moment to reinvigorate the search for a successful contestant.

Turing’s Prescience — The Turing Test and Sophisticated Interaction

Days to Centenary: 253

Every once in a while I’m not looking for Turing, but I find him anyway. Is it just coincidence? Am I simply attuned to seeing reflections of Turing in the world because of working on this blog? I think it is coincidence, and I am attuned in that way, but I suspect that it’s more than that.

I think Turing’s prescience — his ability to form important questions (and hypotheses about the answers to those questions) well in advance of the rest of us — means that his relevance becomes increasingly obvious over time. If I were writing a blog about a prominent alchemist I doubt I would come to see reflections of their theories as prominently in the world around me.

In 1873 Arthur Rimbaud wrote in Une Saison en Enfer (A Season in Hell) that “[i]l faut être absolument moderne,” that is “one must be absolutely modern.” Alan Turing manages this trick better than just about anyone, perhaps in the company of H.G. Wells (and maybe Ray Kurzweill, although more time will have to pass before Ray’s predictive powers can be assessed). Turing was so modern back in 1950, when he devised what we now know as the Turing Test, that he anticipated issues whose dimensions and importance are only now becoming clear to most of us.

All of which leads me to “Milo,” whose TED videos I recently stumbled across. Milo is a homo artificilialis created by researchers at Microsoft as a means of allowing games to be more realistically interactive. His image lives on a screen which you observe, but he’s watching you through a camera as well. His inner workings incorporate elements of artificial intelligence that allow him to react to your body movements, facial expressions, tone of voice and other signals, and in turn control not only his body movements, but his blush response, the dilation of his nostils, and other subtle aspects of his interaction with his environment, including his reactions to you.

The extremely fine detail of Milo’s perception and display mean that not only can he understand crude inputs like those from a keyboard, but he can perceive you at an unprecedented level of detail, improving the fine grain of his input dramatically. And not only can he deliver gross outputs like speaking and moving his body in large-scale ways that give an impression of life, he can output small-scale details approximating the expression of thought and emotion that will evoke in you unconscious reactions that have previously been impossible for less finely tuned simulacra to obtain. All of this adds up to significantly improved believability — Milo excels at “the imitation game,” Turing’s own term for the procedure involved in the Turing Test (for those unfamiliar with the test, there is more on it below).

This BBC article has an embedded video of Milo, while the video below is more recent and addresses public reaction to the first video.

You can interact with Milo with a degree of realism that other artificial humans can’t approach, but does he pass the Turing Test?

In “Computing Machinery and Intelligence,” [pdf] Turing famously postulated a test in which — to summarize his argument at the risk of doing violence to an elegant thought — a human subject interacts through text messages with two participants he or she can’t see, one being another human and the second being a machine which is to be tested for intelligence. If the subject — who can only interact with the participants through text — is unable to correctly identify which participant is the machine, then the machine can be said to have achieved some measure of intelligence.

In other words, if the machine can imitate an intelligent creature with sufficient believability as to make it indistinguishable from an actual intelligent creature, we should treat it as having intelligence. We may not agree that the inner workings that allow it to appear to be thinking are actually capable of producing real thoughts, but the inner workings don’t matter. In an experiential sense we don’t directly confront the inner workings of our friends or families — we deal with their external signals, like words and gestures — and yet we treat them as possessing intelligence.

So what about Milo? Peter Molyneux, who showed off Milo at TED, says “[m]ost of it is just a trick – but it is a trick that actually works.” The thing is that in the Turing Test, if it works, it works — external functionality is the sole standard by which a prospective thinking entity is measured and “tricks” are irrelevant. One of the fundamental questions underlying the Turing Test is: if something can interact with a person in such a way as to appear indistinguishable from that which we accept as intelligent, on what basis do we deny it the label “intelligent”?

As homo artificialis becomes better at imitating homo sapiens, this question is thrown into greater and greater relief. You and I see it when we look at Milo — Turing saw it in his minds eye at a time when a telephone was a heavy bakelite object tethered to a wall by a cord and a computer was something like ENIAC, which filled a room and weighed 30 tons.

Obviously I see Turing in the world around me because, through his pioneering work in mathematics and computing, he is embodied in that world, but very importantly I also see him everywhere because his almost magically prescient thought makes him more and more relevant as time goes on.