Papperoni

2017

RACE: Large-Scale Reading Comprehension Dataset From Examinations

Guokun Lai, Q. Xie, Haozhe Liu, Yining Yang, Eduard Hovy

Open PDF Google Scholar

citations

Cite Score

51

AI summary

This paper introduces RACE, a new reading comprehension dataset with 28,000 passages and 100,000 questions from English exams for Chinese students, evaluating understanding and reasoning, with state-of-the-art models achieving 43% performance compared to human 95% performance.

Main Contributions

Introduces RACE, a large-scale reading comprehension dataset from English exams.
The dataset contains near 28,000 passages and 100,000 questions.
Questions in RACE require more reasoning than other datasets.
The dataset covers a broad range of topics and writing styles.
Achieved 43% performance with state-of-the-art models, highlighting a significant gap compared to human performance (95%).

Abstract

We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students’ ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/˜glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines

Citation Graph

Loading graph...

References [20]

Sort:

Filter:

[1]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[2]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

Very cool idea. Simple yet very impactful!

[3]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[4]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[5]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[6]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[7]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[8]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[9]Embracing Data Abundance: Booktest Dataset for Reading Comprehension

O. Bajgar, R. Kadlec, Jan Kleindienst - 2016

2 papers in library cite

A bit poorly written, throws some shade in other papers, and very marginal contributions (they just perform the same data extraction as others on more data)

[10]MS MARCO: A Human Generated Machine Reading Comprehension Dataset

T. N. Nguyen, M. Rosenberg, X. Song, Jianfeng Gao, S. Tiwary, R. Majumder, L. Deng - 2016

8 papers in library cite

I am not sure if this is very relevant, but it has a few citations.

[11]Text Understanding With the Attention Sum Reader Network

R. Kadlec, M. Schmid, O. Bajgar, Jan Kleindienst - 2016

7 papers in library cite

Not too cited but cited a lot internally

[12]Newsqa: A Machine Comprehension Dataset

A. Trischler, Tianle Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, K. Suleman - 2017

6 papers in library cite

[13]Who Did What: A Large-Scale Person-Centered Cloze Dataset

T. Onishi, Haiming Wang, Mohit Bansal, Kevin Gimpel, D. Mcallester - 2016

4 papers in library cite

[14]Gated-Attention Readers for Text Comprehension

B. Dhingra, Haozhe Liu, W. W. Cohen, Ruslan Salakhutdinov - 2016

3 papers in library cite

[15]Automatic Evaluation of Summaries Using N-Gram Co-Occurrence Statistics

Chin Yew Lin, Eduard Hovy - 2003

2 papers in library cite

[16]Question Answering via Integer Programming Over Semi-Structured Knowledge

Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, Dan Roth - 2016

2 papers in library cite

[17]Overview of Clef Qa Entrance Exams Task 2014

A. Penas, Y. Miyao, A. Rodrigo, Eduard Hovy, N. Kando - 2014

1 paper in library cites

[18]Overview of Clef Qa Entrance Exams Task 2014

A. Penas, Y. Miyao, A. Rodrigo, Eduard Hovy, N. Kando - 2014

1 paper in library cites

[19]Overview of the ntcir-11 Qa-Lab Task

H. Shibuki, K. Sakamoto, Y. Kano, T. Mitamura, M. Ishioroshi, K. Y. Itakura, D. Wang, T. Mori, N. Kando - 2014

1 paper in library cites

[20]Semi-Supervised Qa With Generative Domain-Adaptive Nets

Zhilin Yang, Jiaxi Hu, Ruslan Salakhutdinov, W. W. Cohen - 2017

1 paper in library cites

Cited by

11

papers in your library

Cites

11

papers in your library

Read

on December 26, 2025

I really like the idea of using human tests for testing AI. Also, very nice insige for using chinese tests!

Tags

Paper Aliases

No aliases