Papperoni

2018

Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang

Open PDF Google Scholar

citations

Cite Score

69

AI summary

This paper introduces SQuAD 2.0, a dataset for reading comprehension that combines existing SQuAD data with new unanswerable questions crafted by crowdworkers, challenging models to abstain from answering when no answer is supported, with a strong neural system achieving only 66% F1 on SQuAD 2.0.

Main Contributions

Introduces SQuAD 2.0, a new dataset for reading comprehension that includes unanswerable questions.
The unanswerable questions are written adversarially by crowdworkers to look similar to answerable ones.
SQuAD 2.0 forces models to not only answer questions when possible but also determine when no answer is supported by the paragraph and abstain from answering.
A strong neural system that gets 86% F1 on SQUAD 1.1 achieves only 66% F1 on SQuAD 2.0, showing that SQuAD 2.0 is more challenging.
Analysis of different types of negative examples in SQuAD 2.0 is performed.

Abstract

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQUAD 2.0, the latest version of the Stanford Question Answering Dataset (SQUAD). SQUAD 2.0 combines existing SQUAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQUAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQUAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQUAD 1.1 achieves only 66% F1 on SQuAD 2.0.

Citation Graph

Loading graph...

References [24]

Sort:

Filter:

[1]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[2]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[3]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[4]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[5]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[6]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[7]Adversarial Examples for Evaluating Reading Comprehension Systems

R. Jia, Percy Liang - 2017

11 papers in library cite

I liked it a lot! It's good to see people testing things rather than just trying to beat SotA!

[8]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[9]A Sick Cure for the Evaluation of Compositional Distributional Semantic Models

Marco Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Z. Elli - 2014

7 papers in library cite

Just a basic dataset paper. Nothing much.

[10]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[11]Simple and Effective Multi-Paragraph Reading Comprehension

C. Clark, Matt Gardner - 2017

7 papers in library cite

Very nice paper! I think it's a stretch to call it "simple", but the paper is very well written and easy to follow.

[12]MS MARCO: A Human Generated Machine Reading Comprehension Dataset

T. N. Nguyen, M. Rosenberg, X. Song, Jianfeng Gao, S. Tiwary, R. Majumder, L. Deng - 2016

8 papers in library cite

I am not sure if this is very relevant, but it has a few citations.

[13]Gated Self-Matching Networks for Reading Comprehension and Question Answering

Wenyi Wang, N. Yang, F. Wei, B. Chang, M. Zhou - 2017

5 papers in library cite

QA Model

[14]Zero-Shot Relation Extraction via Reading Comprehension

Omer Levy, M. Seo, E. Choi, L. S. Zettlemoyer - 2017

3 papers in library cite

Seems nice - they answer relations that were never seen in the dataset

[15]Making Neural qa as Simple as Possible but Not Simpler

Dirk Weissenborn, G. Wiese, L. Seiffe - 2017

4 papers in library cite

I love the title

[16]Newsqa: A Machine Comprehension Dataset

A. Trischler, Tianle Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, K. Suleman - 2017

6 papers in library cite

[17]Wikiqa: A Challenge Dataset for Open-Domain Question Answering

Yining Yang, W. T. Yih, C. Meek - 2015

4 papers in library cite

[18]Wikireading: A Novel Large-Scale Language Understanding Task Over Wikipedia

D. Hewlett, A. Lacoste, Llion Jones, Illia Polosukhin, A. Fandrianto, J. Han, M. Kelcey, D. Berthelot - 2016

4 papers in library cite

[19]Position-Aware Attention and Supervised Data Improve Slot Filling

Y. Z. Zhang, Victor Zhong, Deli Chen, G. Angeli, Christopher D. Manning - 2017

3 papers in library cite

[20]Reinforced Mnemonic Reader for Machine Reading Comprehension

M. Hu, Y. Peng, Zhongqiang Huang, X. Qiu, F. Wei, M. Zhou - 2018

3 papers in library cite

[21]Daemo: A Self-Governed Crowdsourcing Marketplace

S. N. Gaikwad, D. Morina, R. Nistala, M. Agarwal, A. Cossette, R. Bhanu, S. Savage, V. Narwal, K. Rajpal, J. Regino - 2015

2 papers in library cite

[22]Fusionnet: Fusing via Fully-Aware Attention With Application to Machine Comprehension

H. Y. Huang, C. Zhu, Y. Shen, Weizhu Chen - 2017

2 papers in library cite

[23]Question Answering Using Enhanced Lexical Semantic Models

W. Yih, M. Chang, C. Meek, A. Pastusiak - 2013

1 paper in library cites

[24]What Is the Jeopardy Model? A Quasi-Synchronous Grammar for QA

Mingliang Wang, Noah A. Smith, T. Mitamura - 2007

1 paper in library cites

Cited by

14

papers in your library

Cites

15

papers in your library

Read

on December 25, 2025

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

Tags

Paper Aliases

No aliases