Papperoni

2015

Show and Tell: A Neural Image Caption Generator

Dumitru Erhan

Open PDF Google Scholar

citations

Cite Score

82

AI summary

This paper introduces a neural image caption generator (NIC) that uses a CNN to encode images and an LSTM to generate descriptive sentences, achieving state-of-the-art results on Pascal, Flickr30k, SBU, and COCO datasets by maximizing the likelihood of the target description sentence given the training image.

Main Contributions

Introduces an end-to-end neural network system (NIC) for generating image captions.
Combines state-of-the-art sub-networks for vision (CNN) and language models (LSTM).
Achieves significantly better performance compared to state-of-the-art approaches on Pascal, Flickr30k, SBU, and COCO datasets.
Demonstrates the effectiveness of transfer learning and data size in image captioning.
Provides an analysis of the learned word embeddings and generation diversity.

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

Citation Graph

Loading graph...

References [33]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[3]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[6]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

Very cool idea. Simple yet very impactful!

[7]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[8]DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

J. Donahue, Y. Jia, Oriol Vinyals, J. Hoffman, N. Zhang, E. Tzeng, Trevor Darrell - 2014

15 papers in library cite

Very nice paper. First I've seen (and based on the text, first ever) about feature extraction for images. It's very nice to see embeddings doing SotA

[9]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[10]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[11]Unifying Visual-Semantic Embeddings With Multimodal Neural Language Models

Richard S. Zemel - 2014

5 papers in library cite

I think what I like the most about this paper is that they can use regularities, but I like that they put the images and text in the same embedding space by increasing similarity of positive samples and reducing similarity of negative ones - neat!

[12]OverFeat: Integrated Recognition, Localization and Detection Using Convolutional Networks

P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, Rob Fergus, Yann Lecun - 2014

16 papers in library cite

Very convoluted method, was SotA for only a bit of time, and the paper is very boring.

[13]Imagenet Large Scale Visual Recognition Challenge

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhongqiang Huang, A. Karpathy, A. Khosla, M. Bernstein - 2014

18 papers in library cite

Imagenet dataset challenge paper

[14]Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

M. Hodosh, P. Young, J. Hockenmaier - 2013

4 papers in library cite

Lays some of the groundwork for the captioning stuff

[15]Multimodal Neural Language Models

R. Kiros, Richard S. Zemel, Ruslan Salakhutdinov - 2014

3 papers in library cite

Early multimodality (first one to use NNs)

[16]Explain Images With Multimodal Recurrent Neural Networks

J. Mao, Weixin Xu, Yining Yang, J. Wang, A. L. Yuille - 2014

3 papers in library cite

Baseline for later captioning, improved on initial multimodel paper

[17]From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference Over Event Descriptions

P. Young, A. L. M. Hodosh, J. Hockenmaier - 2014

5 papers in library cite

[18]Grounded Compositional Semantics for Finding and Describing Images With Sentences

Richard Socher, Quoc Le, C. Manning, A. Ng - 2014

5 papers in library cite

[19]Baby Talk: Understanding and Generating Simple Image Descriptions

G. Kulkarni, V. Premraj, S. Dhar, Shanda Li, Yejin Choi, A. C. Berg, T. L. Berg - 2011

4 papers in library cite

[20]Collective Generation of Natural Image Descriptions

P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, Yejin Choi - 2012

3 papers in library cite

[21]Composing Simple Image Descriptions Using Web-Scale N-Grams

Shanda Li, G. Kulkarni, T. L. Berg, A. C. Berg, Yejin Choi - 2011

3 papers in library cite

[22]Every Picture Tells a Story: Generating Sentences From Images

Ali Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, David Forsyth - 2010

3 papers in library cite

[23]Im2text: Describing Images Using 1 Million Captioned Photographs

V. Ordonez, G. Kulkarni, T. L. Berg - 2011

3 papers in library cite

[24]Midge: Generating Image Descriptions From Computer Vision Detections

M. Mitchell, X. Han, J. Dodge, A. Mensch, A. G. A. P. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, H. D. Iii - 2012

3 papers in library cite

[25]Treetalk : Composition and Compression of Trees for Image Descriptions

P. Kuznetsova, V. Ordonez, T. L. Berg, Yejin Choi - 2014

3 papers in library cite

[26]Collecting Image Annotations Using Amazon's Mechanical Turk

C. Rashtchian, P. Young, M. Hodosh, J. Hockenmaier - 2010

2 papers in library cite

[27]Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

A. Karpathy, Armand Joulin, Li Fei Fei - 2014

2 papers in library cite

[28]Image Description Using Visual Dependency Representations

D. Elliott, F. Keller - 2013

2 papers in library cite

[29]Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

Y. Gong, Lisa Wang, M. Hodosh, J. Hockenmaier, Svetlana Lazebnik - 2014

2 papers in library cite

[30]CIDEr: Consensus-Based Image Description Evaluation

R. Vedantam, C. L. Zitnick, D. Parikh - 2015

1 paper in library cites

[31]Generating Image Descriptions Using Dependency Relational Patterns

A. Aker, R. Gaizauskas - 2010

1 paper in library cites

[32]I2t: Image Parsing to Text Description

B. Z. Yao, X. Yang, L. Lin, M. W. Lee, S. C. Zhu - 2010

1 paper in library cites

[33]Knowledge Representation for the Generation of Quantified Natural Language Descriptions of Vehicle Traffic in Image Sequences

R. Gerber, H. H. Nagel - 1996

1 paper in library cites

Cited by

11

papers in your library

Cites

16

papers in your library

Read

on October 15, 2025

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

Tags

Paper Aliases

No aliases