Papperoni

2016

Embracing Data Abundance: Booktest Dataset for Reading Comprehension

O. Bajgar, R. Kadlec, Jan Kleindienst

Open PDF Google Scholar

citations

Cite Score

4

AI summary

This paper introduces the BookTest dataset, a new dataset for reading comprehension, similar to the Children's Book Test (CBT), but 60 times larger, and shows that training on the new data improves the accuracy of the Attention-Sum Reader model on the original CBT test data.

Main Contributions

Introduces the BookTest dataset, a new dataset for reading comprehension that is 60 times larger than the Children's Book Test (CBT).
Shows that training on the new data improves the accuracy of the Attention-Sum Reader model on the original CBT test data.
Demonstrates that the ensemble exceeds the human baseline on one version of the dataset.
Shows that there is still space for further improvement through a human study.
Highlights the importance of data abundance in improving model performance for text comprehension tasks.

Abstract

There is a practically unlimited amount of natural language data available. Still, recent work in text comprehension has focused on datasets which are small relative to current computing possibilities. This article is making a case for the community to move to larger data and as a step in that direction it is proposing the BookTest, a new dataset similar to the popular Children's Book Test (CBT), however more than 60 times larger. We show that training on the new data improves the accuracy of our Attention-Sum Reader model on the original CBT test data by a much larger margin than many recent attempts to improve the model architecture. On one version of the dataset our ensemble even exceeds the human baseline provided by Facebook. We then show in our own human study that there is still space for further improvement.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[3]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[4]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[5]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[6]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[7]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[8]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[9]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[10]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[11]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[12]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[13]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[14]Text Understanding With the Attention Sum Reader Network

R. Kadlec, M. Schmid, O. Bajgar, Jan Kleindienst - 2016

7 papers in library cite

Not too cited but cited a lot internally

[15]A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories

N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, J. Allen - 2016

5 papers in library cite

[16]Cloze Procedure: A New Tool for Measuring Readability

W. L. Taylor - 1953

4 papers in library cite

[17]Who Did What: A Large-Scale Person-Centered Cloze Dataset

T. Onishi, Haiming Wang, Mohit Bansal, Kevin Gimpel, D. Mcallester - 2016

4 papers in library cite

[18]Wikireading: A Novel Large-Scale Language Understanding Task Over Wikipedia

D. Hewlett, A. Lacoste, Llion Jones, Illia Polosukhin, A. Fandrianto, J. Han, M. Kelcey, D. Berthelot - 2016

4 papers in library cite

[19]Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering

P. L. Li, Wentao Li, Z. He, Xinpeng Wang, Yue Cao, Jingren Zhou, Weixin Xu - 2016

3 papers in library cite

[20]Dynamic Entity Representation With Max-Pooling Improves Machine Reading

S. Kobayashi, R. Tian, N. Okazaki, K. Inui - 2016

3 papers in library cite

[21]Gated-Attention Readers for Text Comprehension

B. Dhingra, Haozhe Liu, W. W. Cohen, Ruslan Salakhutdinov - 2016

3 papers in library cite

[22]Reasonet: Learning to Stop Reading in Machine Comprehension

Y. Shen, P. Huang, Jianfeng Gao, Weizhu Chen - 2017

3 papers in library cite

[23]Scaling to Very Very Large Corpora for Natural Language Disambiguation

M. Banko, E. Brill - 2001

3 papers in library cite

[24]The Unreasonable Effectiveness of Data

A. Halevy, P. Norvig, Fernando Pereira - 2009

3 papers in library cite

[25]Attention-Over-Attention Neural Networks for Reading Comprehension

Y. Cui, Ziru Chen, S. Wei, Shijie Wang, T. Liu, G. Hu - 2016

2 papers in library cite

[26]Attentive Pooling Networks

C. D. Santos, M. Tan, Bing Xiang, B. Zhou - 2016

2 papers in library cite

[27]Feature-Rich Part-of-Speech Tagging With a Cyclic Dependency Network

Kristina Toutanova, Dan Klein, Christopher D. Manning, Yoram Singer - 2003

2 papers in library cite

[28]Iterative Alternating Neural Attention for Machine Reading

A. Sordoni, P. Bachman, Yoshua Bengio - 2016

2 papers in library cite

[29]Natural Language Comprehension With the Epireader

A. Trischler, Z. Ye, X. Yuan, K. Suleman - 2016

2 papers in library cite

[30]A Parallel-Hierarchical Model for Machine Comprehension on Sparse Data

A. Trischler, Z. Ye, X. Yuan, J. He, P. Bachman, K. Suleman - 2016

1 paper in library cites

[31]Blocks and Fuel : Frameworks for Deep Learning

B. V. Merrienboer, D. Bahdanau, V. Dumoulin, D. Serdyuk, D. W. Farley, J. Chorowski, Yoshua Bengio - 2015

1 paper in library cites

[32]Consensus Attention-Based Neural Networks for Chinese Reading Comprehension

Y. Cui, T. Liu, Ziru Chen, Shijie Wang, G. Hu - 2016

1 paper in library cites

[33]Incorporating Non-Local Information Into Information Extraction Systems by Gibbs Sampling

J. R. Finkel, T. Grenager, C. Manning - 2005

1 paper in library cites

[34]Never-Ending Learning

T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, Matt Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, Aman Gupta, X. Chen, A. Saparov, M. Greaves, J. Welling - 2015

1 paper in library cites

[35]Separating Answers From Queries for Neural Reading Comprehension

Dirk Weissenborn - 2016

1 paper in library cites

[36]Some of My Best Friends Are Linguists Applying Information Theoretic Methods : Evaluation of Grammar Quality

Frederick Jelinek - 2004

1 paper in library cites

Cited by

2

papers in your library

Cites

14

papers in your library

Read

on October 31, 2025

A bit poorly written, throws some shade in other papers, and very marginal contributions (they just perform the same data extraction as others on more data)

Tags

Paper Aliases

No aliases