Papperoni

2016

SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang

Open PDF Google Scholar

citations

Cite Score

85

AI summary

This paper introduces SQUAD, a new reading comprehension dataset with 100,000+ questions from crowdworkers on Wikipedia articles, where answers are text segments from the passage. It analyzes required reasoning, builds a logistic regression model achieving 51.0% F1, and highlights the challenge for future research due to high human performance (86.8%).

Main Contributions

Introduces SQUAD, a large-scale reading comprehension dataset with 100,000+ questions and answers.
The dataset requires selecting answers from the passage, unlike previous datasets with answer choices.
Analysis of the dataset reveals diverse question types and reasoning challenges.
A logistic regression model achieves an F1 score of 51.0% on SQuAD.
The dataset is made freely available to encourage further research in reading comprehension.

Abstract

We present the Stanford Question Answering Dataset (SQUAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com.

Citation Graph

Loading graph...

References [27]

Sort:

Filter:

[1]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[2]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[3]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[4]Towards AI-complete Question Answering: A Set of Prerequisite Toy Tasks

Jason Weston, Antoine Bordes, S. Chopra, Tomas Mikolov - 2015

11 papers in library cite

It's a good idea and a nice read but the bad part is that most of the tasks are already easy.

[5]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[6]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[7]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[8]Machine Comprehension Using Match-LSTM and Answer Pointer

Shijie Wang, J. J. Jiang - 2017

6 papers in library cite

Answer pointer

[9]Modeling Biological Processes for Reading Comprehension

Jonathan Berant, Vivek Srikumar, P. C. Chen, B. Huang, B. Manning, Peter Clark - 2014

4 papers in library cite

[10]Wikiqa: A Challenge Dataset for Open-Domain Question Answering

Yining Yang, W. T. Yih, C. Meek - 2015

4 papers in library cite

[11]Building Watson: An Overview of the DeepQA Project

D. Ferrucci, E. Brown, J. C. Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, C. Welty - 2013

3 papers in library cite

[12]Learning to Automatically Solve Algebra Word Problems

Nate Kushman, Y. Artzi, Luke Zettlemoyer, R. Barzilay - 2014

3 papers in library cite

[13]Machine Comprehension With Syntax, Frames, and Semantics

Haiming Wang, Mohit Bansal, Kevin Gimpel, D. Mcallester - 2015

3 papers in library cite

[14]A Rule-Based Question Answering System for Reading Comprehension Tests

E. Riloff, M. Thelen - 2000

2 papers in library cite

[15]Building a Question Answering Test Collection

E. M. Voorhees, D. M. Tice - 2000

2 papers in library cite

[16]Daemo: A Self-Governed Crowdsourcing Marketplace

S. N. Gaikwad, D. Morina, R. Nistala, M. Agarwal, A. Cossette, R. Bhanu, S. Savage, V. Narwal, K. Rajpal, J. Regino - 2015

2 papers in library cite

[17]Deep Read: A Reading Comprehension System

L. Hirschman, M. Light, E. Breck, J. D. Burger - 1999

2 papers in library cite

[18]Learning Answer-Entailing Structures for Machine Comprehension

M. Sachan, A. Dubey, E. P. Xing, M. Richardson - 2015

2 papers in library cite

[19]Learning to Solve Arithmetic Word Problems With Verb Categorization

M. J. Hosseini, Hananneh Hajishirzi, Oren Etzioni, Nate Kushman - 2014

2 papers in library cite

[20]My Computer Is an Honor Student but How Intelligent Is It? Standardized Tests as a Measure of AI

Peter Clark, Oren Etzioni - 2016

2 papers in library cite

[21]A Machine Learning Approach to Answering Questions for Reading Comprehension Tests

H. T. Ng, L. H. Teo, J. L. P. Kwan - 2000

1 paper in library cites

[22]An Analysis of the AskMSR Question-Answering System

E. Brill, S. Dumais, M. Banko - 2002

1 paper in library cites

[23]Answer Extraction From Passage Graph for Question Answering

Huan Sun, N. Duan, Y. Duan, M. Zhou - 2013

1 paper in library cites

[24]Exploring Correlation of Dependency Relation Paths for Answer Extraction

D. Shen, D. Klakow - 2006

1 paper in library cites

[25]Learning Surface Text Patterns for a Question Answering System

D. Ravichandran, Eduard Hovy - 2002

1 paper in library cites

[26]Machine Comprehension With Discourse Relations

K. Narasimhan, R. Barzilay - 2015

1 paper in library cites

[27]N-Gram Idf: A Global Term Weighting Scheme Based on Information Distance

M. Shirakawa, T. Hara, S. Nishio - 2015

1 paper in library cites

Cited by

37

papers in your library

Cites

8

papers in your library

Read

on October 17, 2025

Nice paper that introduced an important dataset. Not much else though.

Tags

Paper Aliases

No aliases