Papperoni

2015

Towards AI-complete Question Answering: A Set of Prerequisite Toy Tasks

Jason Weston, Antoine Bordes, S. Chopra, Tomas Mikolov

Open PDF Google Scholar

citations

Cite Score

44

AI summary

This paper introduces a framework and a set of 20 synthetic tasks, built with a unified underlying simulation of a physical world, for developing learning algorithms for text understanding and reasoning; and it shows that Memory Networks model is able to solve some, but not all, of the tasks.

Main Contributions

Propose a set of proxy tasks that evaluate reading comprehension via question answering.
Tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more.
Tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human.
Classify these tasks into skill sets, so that researchers can identify (and then rectify) the failings of their systems.
Extend and improve the recently introduced Memory Networks model, and show it is able to solve some, but not all, of the tasks.

Abstract

One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. We believe many existing learning systems can currently not solve them, and hence our aim is to classify these tasks into skill sets, so that researchers can identify (and then rectify) the failings of their systems. We also extend and improve the recently introduced Memory Networks model, and show it is able to solve some, but not all, of the tasks.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[5]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[6]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[7]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[8]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[9]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[10]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[11]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[12]Understanding Natural Language

T. Winograd - 1972

7 papers in library cite

Book, but early work on NLU

[13]Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, Victor Zhong, R. Paulus, Richard Socher - 2015

9 papers in library cite

[14]Large-Scale Simple Question Answering With Memory Networks

Antoine Bordes, Nicolas Usunier, S. Chopra, Jason Weston - 2015

5 papers in library cite

Mem networks for QA - sounds interesting

[15]Semantic Parsing on Freebase From Question-Answer Pairs

Jonathan Berant, A. Chou, R. Frostig, Percy Liang - 2013

8 papers in library cite

[16]Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions

X. Zhu, Zoubin Ghahramani, J. Lafferty - 2003

5 papers in library cite

[17]Modeling Biological Processes for Reading Comprehension

Jonathan Berant, Vivek Srikumar, P. C. Chen, B. Huang, B. Manning, Peter Clark - 2014

4 papers in library cite

[18]Perceptron: An Introduction to Computational Geometry

M. Minsky, S. Papert - 1969

4 papers in library cite

[19]Paraphrase-Driven Learning for Open Question Answering

A. Fader, Luke Zettlemoyer, Oren Etzioni - 2013

3 papers in library cite

[20]The Unreasonable Effectiveness of Data

A. Halevy, P. Norvig, Fernando Pereira - 2009

3 papers in library cite

[21]Learning to Interpret Natural Language Navigation Instructions From Observations

Deli Chen, R. Mooney - 2011

2 papers in library cite

[22]Open Question Answering Over Curated and Extracted Knowledge Bases

A. Fader, Luke Zettlemoyer, Oren Etzioni - 2014

2 papers in library cite

[23]Predicting Time Series With Support Vector Machines

K. Muller, A. Smola, G. Ratsch, B. Scholkopf, J. Kohlmorgen, V. Vapnik - 1997

2 papers in library cite

[24]Towards Neural Network-Based Rea soning

B. Peng, Z. L. Lu, H. Li, K. Wong - 2015

2 papers in library cite

[25]Towards Understanding Situated Natural Language

Antoine Bordes, Nicolas Usunier, Ronan Collobert, Jason Weston - 2010

2 papers in library cite

[26]UCI Machine Learning Repository

K. Bache, M. Lichman - 2013

2 papers in library cite

[27]A Machine Learning Approach to Coreference Resolution of noun Phrases

W. Soon, H. Ng, D. Lim - 2001

1 paper in library cites

[28]A Multi-Pass Sieve for Coreference Resolution

K. Raghunathan, Honglak Lee, S. Rangarajan, N. Chambers, M. Surdeanu, Dan Jurafsky, C. Manning - 2010

1 paper in library cites

[29]Factor-Based Compositional Embedding Models

M. Yu, M. Gormley, Mark Dredze - 2014

1 paper in library cites

[30]Freebase qa: Information Extraction or Semantic Parsing?

Xingcheng Yao, Jonathan Berant, B. V. Durme - 2014

1 paper in library cites

[31]Lambda Dependency-Based Compositional Semantics

Percy Liang - 2013

1 paper in library cites

[32]Learning Dependency-Based Compositional Semantics

Percy Liang, M. Jordan, Dan Klein - 2013

1 paper in library cites

[33]On Spectral Clustering: Analysis and an Algo Rithm

A. Ng, M. Jordan, Y. Weiss - 2002

1 paper in library cites

[34]On the Effective Use of Cyc in a Question Answering System

J. Curtis, G. Matthews, D. Baxter - 2005

1 paper in library cites

[35]Tempeval-3: Evaluating Events, Time Expressions, and Temporal Relations

N. Uzzaman, H. Llorens, J. Allen, L. Derczynski, M. Verhagen, J. Pustejovsky - 2012

1 paper in library cites

[36]Twisty Little Passages: An Approach to Interactive Fiction

N. Montfort - 2005

1 paper in library cites

Cited by

11

papers in your library

Cites

14

papers in your library

Read

on November 23, 2025

It's a good idea and a nice read but the bad part is that most of the tasks are already easy.

Tags

Paper Aliases

No aliases