Chatbots - Benchmarking and The Turing Test

Can judges be fooled into thinking a computer is human? 

IMAGE: The Turing Archive

IMAGE: The Turing Archive

copy of article published in Mind (Vol. LIX, Oct. 1950). received from the estate of Mrs Sara Turing

copy of article published in Mind (Vol. LIX, Oct. 1950). received from the estate of Mrs Sara Turing

Alan Turing (1912-54) was a brilliant British mathematician and is best-known for helping decipher the code created by German Enigma machines in the Second World War, and for being one of the founders of computer science and artificial intelligence.

In 1950, his article Computing Machinery and Intelligence appeared in the philosophy journal Mind in which he posed the question "Can a Machine Think?" and if so, how could this be measured?  He suggested that if a computer could give answers that could not be told apart from answers a human may give, then it could be said that the computer was thinking. This field is generally known as natural language processing.

The Turing Test

Alan developed the Turing Test to evaluate if a machine could convince us it was human. The test involved a computer and a person engaging in a scripted typed conversation in real time and the objective was for a human judge to see if they could tell the machine from the human and if they couldn’t then the machine passed the test.

The Imitation Game

The test wasn’t focussed on whether or not the computer had given correct answers to questions but on how closely answers resembled those a human would give. Rather than trying to determine if a machine is thinking, Turing suggests we should ask if the machine can win a game, called the "Imitation Game".

The Loebner Prize

The Loebner Prize organised by The Society for the Study of Artificial Intelligence and Simulation of Behaviour (AISB).

The Loebner Prize organised by The Society for the Study of Artificial Intelligence and Simulation of Behaviour (AISB).

In 1990 Dr Hugh Loebner agreed to underwrite a competition designed to implement the Turing Test withThe Cambridge Centre for Behavioural Studies.

The winner of the annual contest is the best entry relative to other entries that year. There is a cash prize and a medal for first place, with a silver medal and prize for the computer able to fool most of the judges it’s human using text conversation alone. The bronze prize goes to  the computer deemed to be the most “human-like”.

This year (2017) Bletchley Park hosted the annual Loebner Prize.IMAGE: Bletchley Park Org

This year (2017) Bletchley Park hosted the annual Loebner Prize.IMAGE: Bletchley Park Org

The Loeberner Prize is an exciting competition attracting leading edge entries year on year.

The AISB have taken over the running of the contest and their goal is to continue the work based on Turing’s ground-breaking paper and started by Loebner. The mission is to make the competition even more popular and far reaching, not only fuelling global interest and excitement on Artificial Intelligence but also to drive research.

For further information please contact the Loebner Prize Committee on lpc@aisb.org.uk. There is plenty of additional information also on their website.