Papperoni

2017

Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi

Open PDF Google Scholar

citations

Cite Score

59

AI summary

This paper introduces the Bi-Directional Attention Flow (BIDAF) network for machine comprehension, utilizing bi-directional attention flow and hierarchical processing to achieve state-of-the-art results on the SQuAD dataset and CNN/DailyMail cloze test.

Main Contributions

Introduces the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical architecture for machine comprehension.
Employs bi-directional attention flow to obtain a query-aware context representation.
Achieves state-of-the-art results on the Stanford Question Answering Dataset (SQuAD).
Achieves state-of-the-art results on the CNN/DailyMail cloze test.
Provides an in-depth ablation study of the model on the SQUAD development set.

Abstract

Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQUAD) and CNN/DailyMail cloze test.

Citation Graph

Loading graph...

References [33]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[6]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[7]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[8]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[9]Memory Networks

Jason Weston, S. Chopra, Antoine Bordes - 2015

18 papers in library cite

The first half of the paper (when they discuss the concept in a very abstract way) is amazing. However, the actual methodology was very convoluted - I did not like it. I thought that Neural Turing Machines were inspired in this, but actually they are contemporary... So anyway, the concept is nice, execution is not.

[10]MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

M. Richardson, C. J. C. Burges, Erin Renshaw - 2013

16 papers in library cite

Maybe the best dataset paper I have ever read. So well explained, thoroughly thought! It's a shame it's a very small dataset...

[11]The Goldilocks Principle: Reading Children's Books With Explicit Memory Representations

F. Hill, Antoine Bordes, S. Chopra, Jason Weston - 2015

14 papers in library cite

Cool use of memory networks.

[12]A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Deli Chen, J. Bolton, Christopher D. Manning - 2016

9 papers in library cite

Very solid work that shows that things are not always what they seem - very nice!

[13]Vqa: Visual Question Answering

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, D. Parikh - 2015

6 papers in library cite

New task?

[14]Highway Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

Introduced highway networks, which seem like a precursor to resnets

[15]Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

A. Fukui, D. H. Park, Diyi Yang, A. Rohrbach, Trevor Darrell, M. Rohrbach - 2016

2 papers in library cite

Visual QA

[16]Machine Comprehension Using Match-LSTM and Answer Pointer

Shijie Wang, J. J. Jiang - 2017

6 papers in library cite

Answer pointer

[17]Text Understanding With the Attention Sum Reader Network

R. Kadlec, M. Schmid, O. Bajgar, Jan Kleindienst - 2016

7 papers in library cite

Not too cited but cited a lot internally

[18]Dynamic Memory Networks for Visual and Textual Question Answering

Caiming Xiong, S. Merity, Richard Socher - 2016

5 papers in library cite

[19]Dynamic Coattention Networks for Question Answering

Caiming Xiong, Victor Zhong, Richard Socher - 2017

3 papers in library cite

[20]Dynamic Entity Representation With Max-Pooling Improves Machine Reading

S. Kobayashi, R. Tian, N. Okazaki, K. Inui - 2016

3 papers in library cite

[21]End-to-End Reading Comprehension With Dynamic Answer Chunk Ranking

Y. Yu, Wenxuan Zhang, K. S. Hasan, M. Yu, Bing Xiang, B. Zhou - 2016

3 papers in library cite

[22]Gated-Attention Readers for Text Comprehension

B. Dhingra, Haozhe Liu, W. W. Cohen, Ruslan Salakhutdinov - 2016

3 papers in library cite

[23]Learning Recurrent Span Representations for Extractive Question Answering

K. Lee, S. Salant, T. Kwiatkowski, A. P. Parikh, Dipanjan Das, Jonathan Berant - 2017

3 papers in library cite

[24]Reasonet: Learning to Stop Reading in Machine Comprehension

Y. Shen, P. Huang, Jianfeng Gao, Weizhu Chen - 2017

3 papers in library cite

[25]Attention-Over-Attention Neural Networks for Reading Comprehension

Y. Cui, Ziru Chen, S. Wei, Shijie Wang, T. Liu, G. Hu - 2016

2 papers in library cite

[26]Iterative Alternating Neural Attention for Machine Reading

A. Sordoni, P. Bachman, Yoshua Bengio - 2016

2 papers in library cite

[27]Natural Language Comprehension With the Epireader

A. Trischler, Z. Ye, X. Yuan, K. Suleman - 2016

2 papers in library cite

[28]Ask Your Neurons: A Neural-Based Approach to Answering Questions About Images

M. Malinowski, M. Rohrbach, M. Fritz - 2015

1 paper in library cites

[29]Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Hu Xu, K. Saenko - 2016

1 paper in library cites

[30]Hierarchical Question-Image Co-Attention for Visual Question Answering

J. Lu, Jihan Yang, D. Batra, D. Parikh - 2016

1 paper in library cites

[31]Stacked Attention Networks for Image Question Answering

Zhilin Yang, X. He, Jianfeng Gao, L. Deng, A. Smola - 2015

1 paper in library cites

[32]Visual7w: Grounded Question Answering in Images

Yuxuan Zhu, O. Groth, M. S. Bernstein, Li Fei Fei - 2016

1 paper in library cites

[33]Words or Characters? Fine-Grained Gating for Reading Comprehension

Zhilin Yang, B. Dhingra, Y. Yuan, Jiaxi Hu, W. W. Cohen, Ruslan Salakhutdinov - 2016

1 paper in library cites

Cited by

13

papers in your library

Cites

17

papers in your library

Read

on October 24, 2025

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

Tags

Paper Aliases

No aliases