Papperoni

2018

CoQA: A Conversational Question Answering Challenge

Siva Reddy, Deli Chen, Christopher D. Manning

Open PDF Google Scholar

citations

Cite Score

46

AI summary

This paper introduces CoQA, a novel conversational question answering dataset containing 127k questions with answers from 8k conversations across seven domains. It highlights challenges like coreference and pragmatic reasoning, achieving 65.4% F1 score with state-of-the-art models, leaving room for improvement compared to human performance at 88.8%.

Main Contributions

Introduces CoQA, a new dataset for conversational question answering with 127k questions and answers.
Demonstrates the challenges of conversational QA, including coreference and pragmatic reasoning.
Provides an in-depth analysis of CoQA and compares it to existing reading comprehension datasets.
Evaluates strong dialogue and reading comprehension models on CoQA, achieving 65.4% F1 score.
Highlights the gap between machine and human performance, indicating areas for future improvement.

Abstract

Humans gather information through conversations involving a series of interconnected questions and answers. For machines to assist in information gathering, it is therefore essential to enable them to answer conversational questions. We introduce CoQA, a novel dataset for building Conversational Question Answering systems. Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The questions are conversational, and the answers are free-form text with their corresponding evidence highlighted in the passage. We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning. We evaluate strong dialogue and reading comprehension models on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points behind human performance (88.8%), indicating there is ample room for improvement. We present CoQA as a challenge to the community at https://stanfordnlp.github.io/coqa.

Citation Graph

Loading graph...

References [54]

Sort:

Filter:

[1]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[2]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[3]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[4]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[5]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[6]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[7]Get to the Point: Summarization With Pointer-Generator Networks

A. See, P. J. Liu, Christopher D. Manning - 2017

8 papers in library cite

It's a bit of the same thing of the other ones. I am not sure if this was the first or not, but I am getting a bit bored of this

[8]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[9]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[10]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[11]A Neural Conversational Model

Oriol Vinyals, Quoc V. Le - 2015

7 papers in library cite

No new methodology, no measurements, no results. It should be like 4 pages long if they didn't fill in with the conversation logs.

[12]Hierarchical Neural Story Generation

A. Fan, Martha Lewis, Yann Dauphin - 2018

4 papers in library cite

I had higher hopes for this paper. The methodology is nice but at this stage things start to get waaay to complicated

[13]RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy - 2017

11 papers in library cite

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

[14]Towards AI-complete Question Answering: A Set of Prerequisite Toy Tasks

Jason Weston, Antoine Bordes, S. Chopra, Tomas Mikolov - 2015

11 papers in library cite

It's a good idea and a nice read but the bad part is that most of the tasks are already easy.

[15]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[16]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[17]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[18]Vqa: Visual Question Answering

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, D. Parikh - 2015

6 papers in library cite

New task?

[19]Reading Wikipedia to Answer Open-Domain Questions

Deli Chen, Adam Fisch, Jason Weston, Antoine Bordes - 2017

10 papers in library cite

Open Domain QA with wikipedia

[20]OpenNMT: Open-Source Toolkit for Neural Machine Translation

G. Klein, Yoon Kim, Y. Deng, J. Senellart, A. Rush - 2017

4 papers in library cite

Framework that I never heard before...

[21]Personalizing Dialogue Agents: I Have a Dog, Do You Have Pets Too?

S. Zhang, E. Dinan, J. Urbanek, A. Szlam, Douwe Kiela, Jason Weston - 2018

4 papers in library cite

Maybe the first time I have seen "agents" in a title

[22]How Not to Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

C. L. Liu, Ryan Lowe, I. Serban, M. Noseworthy, L. Charlin, J. Pineau - 2016

3 papers in library cite

They discuss how to game metrics

[23]MS MARCO: A Human Generated Machine Reading Comprehension Dataset

T. N. Nguyen, M. Rosenberg, X. Song, Jianfeng Gao, S. Tiwary, R. Majumder, L. Deng - 2016

8 papers in library cite

I am not sure if this is very relevant, but it has a few citations.

[24]Quac: Question Answering in Context

E. Choi, He He, M. Iyyer, M. Yatskar, W. T. Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer - 2018

8 papers in library cite

[25]The Narrative Qa Reading Comprehension Challenge

T. Kocisky, J. Schwarz, Phil Blunsom, C. Dyer, K. M. Hermann, G. Melis, Edward Grefenstette - 2018

4 papers in library cite

They say this is "too hard" - why?

[26]Large-Scale Simple Question Answering With Memory Networks

Antoine Bordes, Nicolas Usunier, S. Chopra, Jason Weston - 2015

5 papers in library cite

Mem networks for QA - sounds interesting

[27]Making Neural qa as Simple as Possible but Not Simpler

Dirk Weissenborn, G. Wiese, L. Seiffe - 2017

4 papers in library cite

I love the title

[28]SearchQA: A New Q&A Dataset Augmented With Context From a Search Engine

M. Dunn, L. Sagun, M. Higgins, V. U. Guney, V. Cirik, Kyunghyun Cho - 2017

5 papers in library cite

[29]Semantic Parsing on Freebase From Question-Answer Pairs

Jonathan Berant, A. Chou, R. Frostig, Percy Liang - 2013

8 papers in library cite

[30]Newsqa: A Machine Comprehension Dataset

A. Trischler, Tianle Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, K. Suleman - 2017

6 papers in library cite

[31]A Diversity-Promoting Objective Function for Neural Conversation Models

Jeffrey Li, M. Galley, Chris Brockett, Jianfeng Gao, B. Dolan - 2016

4 papers in library cite

[32]Incorporating Copying Mechanism in Sequence-to-Sequence Learning

J. Gu, Z. L. Lu, H. Li, V. O. K. Li - 2016

4 papers in library cite

[33]Looking Beyond the Surface: A Challenge Set for Reading Comprehension Over Multiple Sentences

Daniel Khashabi, S. Chaturvedi, M. Roth, Shyam Upadhyay, Dan Roth - 2018

4 papers in library cite

[34]Crowdsourcing Multiple Choice Science Questions

J. Welbl, N. F. Liu, Matt Gardner - 2017

3 papers in library cite

[35]Compositional Semantic Parsing on Semi-Structured Tables

P. Pasupat, Percy Liang - 2015

2 papers in library cite

[36]Constructing Datasets for Multi-Hop Reading Comprehension Across Documents

J. Welbl, P. Stenetorp, Sebastian Riedel - 2018

2 papers in library cite

[37]Generative Deep Neural Networks for Dialogue: A Short Review

I. V. Serban, Ryan Lowe, L. C. Lin, J. Pineau - 2016

2 papers in library cite

[38]The Price of Debiasing Automatic Metrics in Natural Language Evalaution

A. Chaganty, S. Mussmann, Percy Liang - 2018

2 papers in library cite

[39]Visual Dialog

A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, D. Batra - 2017

2 papers in library cite

[40]A Dataset for Document Grounded Conversations

K. Zhou, S. Prabhumoye, A. W. Black - 2018

1 paper in library cites

[41]Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs With a Knowledge Graph

A. Saha, V. Pahuja, M. M. Khapra, K. Sankaranarayanan, S. Chandar - 2018

1 paper in library cites

[42]Dataset and Baselines for Sequential Open Domain Question Answering

A. Elgohary, Changsheng Zhao, J. B. Graber - 2018

1 paper in library cites

[43]Elementary School Science and Math Tests as a Driver for AI: Take the Aristo Challenge!

Peter Clark - 2015

1 paper in library cites

[44]End-to-End Audio Visual Scene-Aware Dialog Using Multimodal Attention-Based Video Features

C. Hori, H. Alamri, J. Wang, G. Winchern, T. Hori, A. Cherian, T. K. Marks, V. Cartillier, R. G. Lopes, A. Das, Others - 2018

1 paper in library cites

[45]Fast and Accurate Reading Comprehension by Combining Self-Attention and Convolution

A. W. Yu, David Dohan, Quoc Le, T. Luong, R. Zhao, K. Chen - 2018

1 paper in library cites

[46]FlowQA: Grasping Flow in History for Conversational Machine Comprehension

H. Y. Huang, E. Choi, W. T. Yih - 2019

1 paper in library cites

[47]Interpretation of Natural Language Rules in Conversational Machine Reading

M. Saeidi, M. Bartolo, P. Lewis, Shivalika Singh, Tim Rocktaschel, M. S. Don, G. Bouchard, Sebastian Riedel - 2018

1 paper in library cites

[48]Multi-Relational Question Answering From Narratives: Machine Reading and Reasoning in Simulated Worlds

I. Labutov, B. Yang, A. Prakash, A. Azaria - 2018

1 paper in library cites

[49]ParlAI: A Dialog Research Software Platform

A. Miller, W. Feng, D. Batra, Antoine Bordes, Adam Fisch, J. Lu, D. Parikh, Jason Weston - 2017

1 paper in library cites

[50]SDNet: Contextualized Attention-Based Deep Network for Conversational Question Answering

C. Zhu, M. Zeng, X. D. Huang - 2018

1 paper in library cites

[51]Search-Based Neural Structured Learning for Sequential Question Answering

M. Iyyer, W. T. Yih, M. W. Chang - 2017

1 paper in library cites

[52]SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge

S. Ostermann, M. Roth, A. Modi, S. Thater, M. Pinkal - 2018

1 paper in library cites

[53]The Stanford CoreNLP

Christopher D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, D. Mcclosky - 2014

1 paper in library cites

[54]The Web as a Knowledge-Base for Answering Complex Questions

A. Talmor, Jonathan Berant - 2018

1 paper in library cites

Cited by

6

papers in your library

Cites

28

papers in your library

Read

on November 11, 2025

It's a fine paper and a solid addition to QA data + NLU.

Tags

Paper Aliases

No aliases