Papperoni

2015

Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom

Open PDF Google Scholar

citations

Cite Score

72

AI summary

This paper introduces a methodology to create large-scale supervised reading comprehension datasets using CNN and Daily Mail news articles. It introduces attention based deep neural networks for reading comprehension, achieving good results on the new datasets.

Main Contributions

Introduces a methodology for creating large scale supervised reading comprehension datasets.
Presents two new corpora of roughly a million news stories with associated queries from the CNN and Daily Mail websites.
Develops attention-based deep learning models for reading comprehension.
Shows that attention is a key ingredient for machine reading and question answering due to the need to propagate information over long distances.
Achieves good results on the new datasets.

Abstract

Teaching machines to read natural language documents remains an elusive challenge. Machine reading systems can be tested on their ability to answer questions posed on the contents of documents that they have seen, but until now large scale training and test datasets have been missing for this type of evaluation. In this work we define a new methodology that resolves this bottleneck and provides large scale supervised reading comprehension data. This allows us to develop a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure.

Citation Graph

Loading graph...

References [20]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[5]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[6]Recurrent Models of Visual Attention

V. Mnih, N. Heess, Alex Graves - 2014

5 papers in library cite

It's not as good as the other paper (DRAW), but it's a precursor and it's so nice how the model learns to pay attention. Also very nice to see RL in the mix, and see the possible usages in games and other things.

[7]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[8]DRAW: A Recurrent Neural Network for Image Generation

K. Gregor, Ivo Danihelka, Alex Graves, D. J. Rezende, Daan Wierstra - 2015

5 papers in library cite

This is SO cool! So interesting that it was biologically inspired. First actual image generation that seems to work, also a jump from the RBM/Autoencoder stuff, and the incremental drawing is amazing!

[9]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[10]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[11]Understanding Natural Language

T. Winograd - 1972

7 papers in library cite

Book, but early work on NLU

[12]Supervised Sequence Labelling With Recurrent Neural Networks

Alex Graves - 2012

6 papers in library cite

[13]Cloze Procedure: A New Tool for Measuring Readability

W. L. Taylor - 1953

4 papers in library cite

[14]Lecture 6.5 - Rmsprop: Divide the Gradient by a Running Average of Its Recent Magnitude

T. Tieleman, Geoffrey Hinton - 2012

4 papers in library cite

[15]A Rule-Based Question Answering System for Reading Comprehension Tests

E. Riloff, M. Thelen - 2000

2 papers in library cite

[16]Automatic Generation of Story Highlights

K. Woodsend, Mirella Lapata - 2010

1 paper in library cites

Missing year

[17]Enhancing Single-Document Summarization by Combining RankNet and Third-Party Sources

K. Svore, L. Vanderwende, C. Burges

1 paper in library cites

[18]Frame-Semantic Parsing

Dipanjan Das, Deli Chen, A. F. T. Martins, N. Schneider, Noah A. Smith - 2013

1 paper in library cites

[19]Machine Reading at the University of Washington

H. Poon, J. Christensen, P. Domingos, Oren Etzioni, R. Hoffmann, C. Kiddon, T. Lin, X. Ling, Mausam, A. Ritter, S. Schoenmackers, S. Soderland, D. Weld, F. Wu, Chiyuan Zhang - 2010

1 paper in library cites

[20]Semantic Frame Identification With Distributed Word Representations

K. M. Hermann, Dipanjan Das, Jason Weston, K. Ganchev - 2014

1 paper in library cites

Cited by

31

papers in your library

Cites

11

papers in your library

Read

on August 9, 2025

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

Tags

Paper Aliases

No aliases