Papperoni

2016

The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez

Open PDF Google Scholar

citations

Cite Score

33

AI summary

This paper introduces the LAMBADA dataset for evaluating language understanding through word prediction, requiring models to track information in the broader discourse; it includes 2662 novels of raw text for training language models, but none of several state-of-the-art language models reaches accuracy above 1%.

Main Contributions

Introduces the LAMBADA dataset, a new benchmark for evaluating language models on their ability to understand broad context in text
Demonstrates that existing state-of-the-art language models perform poorly on the LAMBADA dataset, achieving less than 1% accuracy
Provides an analysis of the LAMBADA dataset, highlighting the linguistic phenomena that make it challenging for language models
Shows that LAMBADA requires to capture non-local phenomena for good performance

Abstract

We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.

Citation Graph

Loading graph...

References [22]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[4]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[5]Srilm - An Extensible Language Modeling Toolkit

Andreas Stolcke - 2002

13 papers in library cite

Toolkit for N-grams. Not too relevant and sounds veeeery simple (sorry for those who implemented it). It's nice to see early implementation of OOP though. The paper is boring and doesn't really say much about the framework, more of a description of how to use the commands and n-gram models.

[6]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[7]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[8]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[9]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[10]A Neural Conversational Model

Oriol Vinyals, Quoc V. Le - 2015

7 papers in library cite

No new methodology, no measurements, no results. It should be like 4 pages long if they didn't fill in with the conversation logs.

[11]Towards AI-complete Question Answering: A Set of Prerequisite Toy Tasks

Jason Weston, Antoine Bordes, S. Chopra, Tomas Mikolov - 2015

11 papers in library cite

It's a good idea and a nice read but the bad part is that most of the tasks are already easy.

[12]Reasoning About Entailment With Neural Attention

Tim Rocktaschel, Edward Grefenstette, K. Hermann, T. Kocisky, Phil Blunsom - 2016

5 papers in library cite

It's nice that they are SotA on top of SNLI, but they just apply existing methodologies.

[13]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[14]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[15]A Neural Network Approach to Context-Sensitive Generation of Conversational Responses

A. Sordoni, M. Galley, Michael Auli, Chris Brockett, Yangfeng Ji, M. Mitchell, J. Y. Nie, Jianfeng Gao, B. Dolan - 2015

4 papers in library cite

Generating conversational responses

[16]Learning Longer Memory in Recurrent Neural Networks

Tomas Mikolov, Armand Joulin, S. Chopra, M. Mathieu, Marc'aurelio Ranzato - 2015

8 papers in library cite

RNNs + longer memory

[17]The microsoft research Sentence Completion Challenge

Geoffrey Zweig, C. J. Burges - 2011

6 papers in library cite

[18]Larger-Context Language Modelling

Tianle Wang, Kyunghyun Cho - 2015

4 papers in library cite

[19]Document Context Language Models

Yangfeng Ji, T. Cohn, L. Kong, C. Dyer, J. Eisenstein - 2015

3 papers in library cite

[20]RNNLM - Recurrent Neural Network Language Modeling Toolkit

Tomas Mikolov, S. Kombrink, A. Deoras, Lukas Burget, Jan Cernocky - 2011

2 papers in library cite

[21]Multi-GranCNN: An Architecture for General Matching of Text Chunks on Multiple Levels of Granularity

W. Yin, Hinrich Schutze - 2015

1 paper in library cites

[22]Using Neural Networks for Modelling and Representing Natural Languages

Tomas Mikolov - 2014

1 paper in library cites

Cited by

12

papers in your library

Cites

16

papers in your library

Read

on October 31, 2025

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

Tags

Paper Aliases

No aliases