Papperoni

2015

Show, Attend and Tell: Neural Image Caption Generation With Visual Attention

K. Xu, Jimmy Lei Ba, R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R. Zemel, Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

88

AI summary

This paper introduces an attention-based model that learns to describe the content of images, using convolutional and recurrent neural networks. It uses Flickr8k, Flickr30k and MS COCO datasets. The model achieves state-of-the-art performance on three benchmark datasets.

Main Contributions

Introduces two attention-based image caption generators under a common framework: a deterministic attention mechanism and a stochastic attention mechanism.
Shows how to gain insight and interpret the results of this framework by visualizing where and what the attention focused on.
Validates the usefulness of attention in caption generation with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.

Abstract

Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[4]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[5]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[6]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[7]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[8]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[9]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[10]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[11]Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

R. Williams - 1992

11 papers in library cite

It's alright for formalizing the concept, but it's a bit boring and doesn't add a lot from the middle on. Focuses too much in reviewing existing techniques and in stochastic units.

[12]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[13]Recurrent Models of Visual Attention

V. Mnih, N. Heess, Alex Graves - 2014

5 papers in library cite

It's not as good as the other paper (DRAW), but it's a precursor and it's so nice how the model learns to pay attention. Also very nice to see RL in the mix, and see the possible usages in games and other things.

[14]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[15]Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Alon Lavie - 2014

2 papers in library cite

So boring! Maybe the original Meteor paper is better though

[16]Unifying Visual-Semantic Embeddings With Multimodal Neural Language Models

Richard S. Zemel - 2014

5 papers in library cite

I think what I like the most about this paper is that they can use regularities, but I like that they put the images and text in the same embedding space by increasing similarity of positive samples and reducing similarity of negative ones - neat!

[17]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[18]How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2013

7 papers in library cite

Very interesting despite not being too relevant. Good read and a new way of thinking about RNNs.

[19]Learning Generative Models With Visual Attention

Ruslan Salakhutdinov - 2014

2 papers in library cite

Waaaay to mathy and TBH I think there aren't a lot of good results. But also I don't have a ton of knowledge on generative models, so I may be biased.

[20]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[21]Imagenet Large Scale Visual Recognition Challenge

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhongqiang Huang, A. Karpathy, A. Khosla, M. Bernstein - 2014

18 papers in library cite

Imagenet dataset challenge paper

[22]Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

J. Donahue, L. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, Trevor Darrell - 2014

4 papers in library cite

Early multimodality

[23]Deep Visual-Semantic Alignments for Generating Image Descriptions

A. Karpathy, Li Fei Fei - 2014

6 papers in library cite

Early multimodality

[24]Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

M. Hodosh, P. Young, J. Hockenmaier - 2013

4 papers in library cite

Lays some of the groundwork for the captioning stuff

[25]From Captions to Visual Concepts and Back

H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, Piotr Dollar, Jianfeng Gao, X. He, M. Mitchell, J. Platt - 2014

1 paper in library cites

Early multimodality

[26]Multiple Object Recognition With Visual Attention

Jimmy Lei Ba, V. Mnih, Koray Kavukcuoglu - 2014

4 papers in library cite

Cited by DRAW

[27]Multimodal Neural Language Models

R. Kiros, Richard S. Zemel, Ruslan Salakhutdinov - 2014

3 papers in library cite

Early multimodality (first one to use NNs)

[28]Practical Bayesian Optimization of Machine Learning Algorithms

J. Snoek, Hugo Larochelle, R. P. Adams - 2012

9 papers in library cite

[29]From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference Over Event Descriptions

P. Young, A. L. M. Hodosh, J. Hockenmaier - 2014

5 papers in library cite

[30]Baby Talk: Understanding and Generating Simple Image Descriptions

G. Kulkarni, V. Premraj, S. Dhar, Shanda Li, Yejin Choi, A. C. Berg, T. L. Berg - 2011

4 papers in library cite

[31]Learning to Combine Foveal Glimpses With a Third-Order Boltzmann Machine

Hugo Larochelle, Geoffrey E. Hinton - 2010

4 papers in library cite

[32]Learning Where to Attend With Deep Architectures for Image Tracking

M. Denil, L. Bazzani, Hugo Larochelle, N. D. Freitas - 2012

4 papers in library cite

[33]Collective Generation of Natural Image Descriptions

P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, Yejin Choi - 2012

3 papers in library cite

[34]Composing Simple Image Descriptions Using Web-Scale N-Grams

Shanda Li, G. Kulkarni, T. L. Berg, A. C. Berg, Yejin Choi - 2011

3 papers in library cite

[35]Deep Captioning With Multimodal Recurrent Neural Networks (M-Rnn)

J. Mao, Weixin Xu, Yining Yang, J. Wang, A. Yuille - 2014

3 papers in library cite

[36]Midge: Generating Image Descriptions From Computer Vision Detections

M. Mitchell, X. Han, J. Dodge, A. Mensch, A. G. A. P. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, H. D. Iii - 2012

3 papers in library cite

[37]Treetalk : Composition and Compression of Trees for Image Descriptions

P. Kuznetsova, V. Ordonez, T. L. Berg, Yejin Choi - 2014

3 papers in library cite

[38]Corpus-Guided Sentence Generation of Natural Images

Yining Yang, C. L. Teo, H. D. Iii, Y. Aloimonos - 2011

2 papers in library cite

[39]Image Description Using Visual Dependency Representations

D. Elliott, F. Keller - 2013

2 papers in library cite

[40]The Dynamic Representation of Scenes

R. A. Rensink - 2000

2 papers in library cite

[41]Control of Goal-Directed and Stimulus-Driven Attention in the Brain

M. Corbetta, G. Shulman - 2002

1 paper in library cites

[42]Input Warping for Bayesian Optimization of Non-Stationary Functions

J. Snoek, K. Swersky, R. Zemel, R. A. Adams - 2014

1 paper in library cites

[43]Learning a Recurrent Visual Representation for Image Caption Generation

X. Chen, C. Zitnick - 2014

1 paper in library cites

[44]Lecture 6.5 - Rmsprop

T. Tieleman, Geoffrey Hinton - 2012

1 paper in library cites

[45]The Dropout Learning algorithm

P. Baldi, P. Sadowski - 2014

1 paper in library cites

[46]The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

L. Weaver, N. Tao - 2001

1 paper in library cites

Cited by

12

papers in your library

Cites

27

papers in your library

Read

on August 3, 2025

It's a nice paper. I liked the soft attention way more than the hard one, and I am a bit mad that it wasn't the best lol And also it's the first paper I read about multimodality, but it seems that this was bustling at the time. Also results are kinda bad.

Tags

Paper Aliases

No aliases