Papperoni

2011

The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern

Open PDF Google Scholar

citations

Cite Score

53

AI summary

This paper introduces the Winograd Schema Challenge as an alternative to the Turing Test, using pairs of sentences with referential ambiguities. It highlights the use of world knowledge and reasoning abilities to resolve ambiguities obvious to humans but difficult for machines.

Main Contributions

Introduces the Winograd Schema Challenge as a new type of Turing Test.
WS challenge depends on binary questions and appeals to world knowledge and default reasoning abilities, but without depending on an explicit notion of entailment.
WS are designed to be easily disambiguated by human readers but difficult for machines relying on selectional restrictions or statistical techniques.
Provides a library of Winograd Schema questions in a standard format.
Discusses the potential for incremental progress and domain-specific applications within the WS challenge.

Abstract

In this paper, we present an alternative to the Turing Test that has some conceptual and practical advantages. A Wino grad schema is a pair of sentences that differ only in one or two words and that contain a referential ambiguity that is re solved in opposite directions in the two sentences. We have compiled a collection of Winograd schemas, designed so that the correct answer is obvious to the human reader, but can not easily be found using selectional restrictions or statistical techniques over text corpora. A contestant in the Winograd Schema Challenge is presented with a collection of one sen tence from each pair, and required to achieve human-level accuracy in choosing the correct disambiguation.

Citation Graph

Loading graph...

References [28]

Sort:

Filter:

[1]ELIZA - A Computer Program for the Study of Natural Language Communication Between Man and Machine

Joseph Weizenbaum - 1966

4 papers in library cite

Very boring. Most of the paper is just discussing how they created ELIZA (which is very straightforward and not amazing at all). The discussion is very superficial.

[2]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[3]Programs With Common Sense

Mccarthy - 1959

2 papers in library cite

It's very early work on artificial intelligence but this focuses much more in putting knowledge in a computer and making it reason vs. AI.

[4]CAPTCHA: Using Hard AI Problems for Security

John Langford - 2000

2 papers in library cite

The idea for captcha is great, but I feel that the paper is too long (18 pages), and they try to theorize too much in having proofs and etc. Overall a bad experience reading it, and I didn't end up knowing more about captcha.

[5]Turing Test Considered Harmful

Ford, Hayes - 1995

2 papers in library cite

It's a good read but it's terrible arguments!

[6]Machine Reading

Etzioni, Banko, Cafarella - 2006

1 paper in library cites

It's not bad, it's just uninteresting and doesn't really bring anything new.

[7]Why the Turing Test Is AI's Biggest Blind Alley

Blay Whitby - 1997

2 papers in library cite

Very honest and a good discussion however, I think he makes wrong conclusions.

[8]If Not Turing's Test, Then What?

Paul Cohen - 2005

2 papers in library cite

It's a very lucid paper, but no hot takes either. It's vanilla.

[9]Is It Enough to Get the Behaviour Right?

Levesque - 2009

2 papers in library cite

Amazing once again. Levesque is amazingly lucid and understands the real arguments here.

[10]Understanding Natural Language

T. Winograd - 1972

7 papers in library cite

Book, but early work on NLU

[11]Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

M. Roemmele, C. A. Bejan, A. S. Gordon - 2011

4 papers in library cite

[12]Foundations of Statistical Natural Language Processing

Manning, Schutze - 1999

4 papers in library cite

[13]A Framework for Computational Semantics (fracas)

R. Cooper, D. Crouch, J. Eijckl, C. Fox, J. Genabith, J. Japars, H. Kamp, D. Milward, M. Pinkal, M. Poesio - 1996

2 papers in library cite

[14]Can Machines Think?

Dennett - 1998

2 papers in library cite

[15]Knowledge Representation and Reasoning

Brachman, Levesque - 2004

2 papers in library cite

[16]Lessons From a Restricted Turing Test

Shieber - 1994

2 papers in library cite

[17]Precision-Focussed Textual Inference

Bobrow, Condoravdi, Crouch, D. Paiva, Karttunen, King, Mairn, Price, Zaenen - 2007

2 papers in library cite

[18]A Study of Textual Entailment

Rus, Mccarthy, Mcnamara, Graesser - 2007

1 paper in library cites

[19]Computation and Cognition: Toward a Foundation for Cognitive Science

Pylyshyn - 1984

1 paper in library cites

[20]Lexical Based Text Entailment System for Main Task of RTE6

Majumdar, Bhattacharyya - 2010

1 paper in library cites

[21]Measuring Universal Intelligence: Toward an Anytime Intelligence Test

H. Orallo, Dowe - 2010

1 paper in library cites

[22]Mind vs Machine

Christian - 2011

1 paper in library cites

[23]Nonmonotonic Logic and Temporal Projection

Hanks, Mcdermott - 1987

1 paper in library cites

[24]On Intelligence

Hawkins, Blakeslee - 2004

1 paper in library cites

[25]Qualitative Spatial Reasoning in Interpreting Text and Narrative

Davis - 2012

1 paper in library cites

[26]The DARPA Machine Reading Program - Encouraging Linguistic and Reasoning Research With a Series of Reading Tasks

Strassel, Adams, Goldberg, Herr, Keesing, Oblinger, Simpson, Schrag, Wright - 2010

1 paper in library cites

[27]Thinking, Fast and Slow

Kahneman - 2011

1 paper in library cites

[28]Web-Based Models for Natural Language Processing

Lapata, Keller - 2005

1 paper in library cites

Cited by

13

papers in your library

Cites

10

papers in your library

Read

on November 27, 2025

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

Tags

Paper Aliases

No aliases