Papperoni

2017

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer

Open PDF Google Scholar

citations

Cite Score

67

AI summary

This paper introduces TriviaQA, a large-scale reading comprehension dataset with 650K question-answer-evidence triples from trivia enthusiasts, showing complexity, syntactic variability, and the need for multi-sentence reasoning, with baseline models underperforming human accuracy.

Main Contributions

Introduces TriviaQA, a dataset with over 650K question-answer-evidence triples, offering a new resource for reading comprehension models.
Demonstrates that TriviaQA contains complex questions with syntactic and lexical variability.
Shows that TriviaQA requires multi-sentence reasoning.
Presents a manual analysis of the dataset quality and the challenges involved.
Provides baseline experiments demonstrating that TriviaQA is not easily solved.

Abstract

We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study.

Citation Graph

Loading graph...

References [33]

Sort:

Filter:

[1]Show, Attend and Tell: Neural Image Caption Generation With Visual Attention

K. Xu, Jimmy Lei Ba, R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R. Zemel, Yoshua Bengio - 2015

12 papers in library cite

It's a nice paper. I liked the soft attention way more than the hard one, and I am a bit mad that it wasn't the best lol And also it's the first paper I read about multimodality, but it seems that this was bustling at the time. Also results are kinda bad.

[2]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[3]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[4]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[5]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[6]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[7]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[8]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[9]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[10]MS MARCO: A Human Generated Machine Reading Comprehension Dataset

T. N. Nguyen, M. Rosenberg, X. Song, Jianfeng Gao, S. Tiwary, R. Majumder, L. Deng - 2016

8 papers in library cite

I am not sure if this is very relevant, but it has a few citations.

[11]Large-Scale Simple Question Answering With Memory Networks

Antoine Bordes, Nicolas Usunier, S. Chopra, Jason Weston - 2015

5 papers in library cite

Mem networks for QA - sounds interesting

[12]SearchQA: A New Q&A Dataset Augmented With Context From a Search Engine

M. Dunn, L. Sagun, M. Higgins, V. U. Guney, V. Cirik, Kyunghyun Cho - 2017

5 papers in library cite

[13]Semantic Parsing on Freebase From Question-Answer Pairs

Jonathan Berant, A. Chou, R. Frostig, Percy Liang - 2013

8 papers in library cite

[14]Newsqa: A Machine Comprehension Dataset

A. Trischler, Tianle Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, K. Suleman - 2017

6 papers in library cite

[15]Who Did What: A Large-Scale Person-Centered Cloze Dataset

T. Onishi, Haiming Wang, Mohit Bansal, Kevin Gimpel, D. Mcallester - 2016

4 papers in library cite

[16]Wikiqa: A Challenge Dataset for Open-Domain Question Answering

Yining Yang, W. T. Yih, C. Meek - 2015

4 papers in library cite

[17]A Neural Network for Factoid Question Answering Over Paragraphs

M. Iyyer, J. B. Graber, L. Claudino, Richard Socher - 2014

3 papers in library cite

[18]Building Watson: An Overview of the DeepQA Project

D. Ferrucci, E. Brown, J. C. Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, C. Welty - 2013

3 papers in library cite

[19]Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering

P. L. Li, Wentao Li, Z. He, Xinpeng Wang, Yue Cao, Jingren Zhou, Weixin Xu - 2016

3 papers in library cite

[20]Machine Comprehension With Syntax, Frames, and Semantics

Haiming Wang, Mohit Bansal, Kevin Gimpel, D. Mcallester - 2015

3 papers in library cite

[21]Adapting Boosting for Information Retrieval Measures

Q. Wu, C. J. Burges, K. M. Svore, Jianfeng Gao - 2010

2 papers in library cite

[22]Building a Question Answering Test Collection

E. M. Voorhees, D. M. Tice - 2000

2 papers in library cite

[23]Compositional Semantic Parsing on Semi-Structured Tables

P. Pasupat, Percy Liang - 2015

2 papers in library cite

[24]Open Question Answering Over Curated and Extracted Knowledge Bases

A. Fader, Luke Zettlemoyer, Oren Etzioni - 2014

2 papers in library cite

[25]´ Opponent Modeling in Deep Reinforcement Learning

He He, J. B. Graber, K. Kwok, H. D. Iii - 2016

1 paper in library cites

[26]Besting the Quiz master: Crowdsourcing Incremental Classification Games

J. B. Graber, B. Satinoff, He He, H. D. Iii - 2012

1 paper in library cites

[27]Hierarchical Attention Networks for Document Classification

Zhilin Yang, Diyi Yang, C. Dyer, X. He, A. Smola, Eduard Hovy - 2016

1 paper in library cites

[28]Knowledge Graph and Corpus Driven Segmentation and Answer Inference for Telegraphic Entity Seeking Queries

M. Joshi, U. Sawant, S. Chakrabarti - 2014

1 paper in library cites

[29]Knowledge-Based Weak Supervision for Information Extraction of Overlapping Relations

R. Hoffmann, Chiyuan Zhang, X. Ling, Luke Zettlemoyer, D. S. Weld - 2011

1 paper in library cites

[30]Large-Scale Semantic Parsing via Schema matching and Lexicon Extension

Q. Cai, A. Yates - 2013

1 paper in library cites

[31]Learning Joint Query Interpretation and Response Ranking

U. Sawant, S. Chakrabarti - 2013

1 paper in library cites

[32]Modeling Relations and Their Mentions Without Labeled Text

Sebastian Riedel, L. Yao, Andrew Mccallum - 2010

1 paper in library cites

[33]Tagme: On-the-Fly Annotation of Short Text Fragments (By Wikipedia Entities)

P. Ferragina, U. Scaiella - 2010

1 paper in library cites

Cited by

18

papers in your library

Cites

12

papers in your library

Read

on December 25, 2025

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

Tags

Paper Aliases

No aliases