Papperoni

2014

Unifying Visual-Semantic Embeddings With Multimodal Neural Language Models

Richard S. Zemel

Open PDF Google Scholar

citations

Cite Score

51

AI summary

This paper introduces a novel encoder-decoder pipeline that learns a multimodal joint embedding space with images and text and a structure-content neural language model (SC-NLM) for decoding. The results achieve state-of-the-art performance on Flickr8K and Flickr30K using LSTM to encode sentences.

Main Contributions

Introduces an encoder-decoder pipeline for multimodal learning.
Proposes a structure-content neural language model (SC-NLM) that disentangles sentence structure from content.
Achieves state-of-the-art performance on Flickr8K and Flickr30K datasets without object detections.
Demonstrates multimodal regularities using linear encoders.
Shows sample captions generated for 800 images.

Abstract

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[5]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[6]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[7]Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Ross Girshick, J. Donahue, Trevor Darrell, Jitendra Malik - 2014

18 papers in library cite

Good results, beat overfeat, used pretraining for improving performance. Only issue is that the paper is overly long...

[8]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[9]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[10]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[11]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[12]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[13]Linguistic Regularities in Continuous Space Word Representations

Tomas Mikolov, W. T. Yih, Geoffrey Zweig - 2013

8 papers in library cite

This builds on Bengio's "Word Representations" paper to show that embeddings can be used to describe syntactic and semantic characteristics of words. Good contribution, but not sure if too impactful.

[14]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[15]Hybrid Speech Recognition With Deep Bidirectional LSTM

Alex Graves, Navdeep Jaitly, Abdel Rahman Mohamed - 2013

2 papers in library cite

I think this is one of the first Deep Bidirectional LSTMs usages. Good but not too relevant right now.

[16]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[17]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[18]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

Simple and very nice that it showed that NNs can stand on their own for NMT.

[19]Large Scale Image Annotation: Learning to Rank With Joint Word-Image Embeddings

Jason Weston, Samy Bengio, Nicolas Usunier - 2010

3 papers in library cite

Very simple approach! Seems like an early word2vec. I like the joint embeddings as well. However, results are underwhelming.

[20]Devise: A Deep Visual-Semantic Embedding Model

Andrea Frome, G. S. Corrado, J. Shlens, Samy Bengio, Jeffrey Dean, Tomas Mikolov, Marc'aurelio Ranzato - 2013

4 papers in library cite

Identify objects

[21]Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

M. Hodosh, P. Young, J. Hockenmaier - 2013

4 papers in library cite

Lays some of the groundwork for the captioning stuff

[22]Multimodal Neural Language Models

R. Kiros, Richard S. Zemel, Ruslan Salakhutdinov - 2014

3 papers in library cite

Early multimodality (first one to use NNs)

[23]Explain Images With Multimodal Recurrent Neural Networks

J. Mao, Weixin Xu, Yining Yang, J. Wang, A. L. Yuille - 2014

3 papers in library cite

Baseline for later captioning, improved on initial multimodel paper

[24]Multilingual Distributed Representations Without Word Alignment

K. M. Hermann, Phil Blunsom - 2014

3 papers in library cite

They embed the sentence, but can't get a translation directly

[25]A Novel Connectionist System for Unconstrained Handwriting Recognition

Alex Graves, M. Liwicki, Santiago Fernandez, R. Bertolami, H. Bunke, Jürgen Schmidhuber - 2009

5 papers in library cite

[26]From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference Over Event Descriptions

P. Young, A. L. M. Hodosh, J. Hockenmaier - 2014

5 papers in library cite

[27]Grounded Compositional Semantics for Finding and Describing Images With Sentences

Richard Socher, Quoc Le, C. Manning, A. Ng - 2014

5 papers in library cite

[28]Baby Talk: Understanding and Generating Simple Image Descriptions

G. Kulkarni, V. Premraj, S. Dhar, Shanda Li, Yejin Choi, A. C. Berg, T. L. Berg - 2011

4 papers in library cite

[29]Collective Generation of Natural Image Descriptions

P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, Yejin Choi - 2012

3 papers in library cite

[30]Composing Simple Image Descriptions Using Web-Scale N-Grams

Shanda Li, G. Kulkarni, T. L. Berg, A. C. Berg, Yejin Choi - 2011

3 papers in library cite

[31]Every Picture Tells a Story: Generating Sentences From Images

Ali Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, David Forsyth - 2010

3 papers in library cite

[32]Factored 3-Way Restricted Boltzmann machines for Modeling Natural Images

Alex Krizhevsky, Geoffrey E. Hinton - 2010

3 papers in library cite

[33]Im2text: Describing Images Using 1 Million Captioned Photographs

V. Ordonez, G. Kulkarni, T. L. Berg - 2011

3 papers in library cite

[34]Midge: Generating Image Descriptions From Computer Vision Detections

M. Mitchell, X. Han, J. Dodge, A. Mensch, A. G. A. P. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, H. D. Iii - 2012

3 papers in library cite

[35]Treetalk : Composition and Compression of Trees for Image Descriptions

P. Kuznetsova, V. Ordonez, T. L. Berg, Yejin Choi - 2014

3 papers in library cite

[36]Corpus-Guided Sentence Generation of Natural Images

Yining Yang, C. L. Teo, H. D. Iii, Y. Aloimonos - 2011

2 papers in library cite

[37]Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

A. Karpathy, Armand Joulin, Li Fei Fei - 2014

2 papers in library cite

[38]Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

Y. Gong, Lisa Wang, M. Hodosh, J. Hockenmaier, Svetlana Lazebnik - 2014

2 papers in library cite

[39]Multilingual Models for Compositional Distributional Semantics

K. M. Hermann, Phil Blunsom - 2014

2 papers in library cite

[40]Multimodal Deep Learning

J. Ngiam, A. Khosla, M. Kim, J. Nam, Honglak Lee, A. Ng - 2011

2 papers in library cite

[41]Multimodal Learning With Deep Boltzmann Machines

N. Srivastava, Ruslan Salakhutdinov - 2012

2 papers in library cite

[42]Unsupervised Learning of Image Transformations

R. Memisevic, Geoffrey Hinton - 2007

2 papers in library cite

[43]A Deep Architecture for Semantic Parsing

Phil Blunsom, N. D. Freitas, Edward Grefenstette, K. M. Hermann - 2014

1 paper in library cites

[44]A Multiplicative Model for Learning Distributed Text-Based Attribute Representations

R. Kiros, Richard S. Zemel, Ruslan Salakhutdinov - 2014

1 paper in library cites

[45]Learning Cross-Modality Similarity for Multinomial Data

Y. Jia, M. Salzmann, Trevor Darrell - 2011

1 paper in library cites

[46]Translating Video Content to Natural Language Descriptions

M. Rohrbach, W. Qiu, I. Titov, S. Thater, M. Pinkal, B. Schiele - 2013

1 paper in library cites

Cited by

5

papers in your library

Cites

24

papers in your library

Read

on October 13, 2025

I think what I like the most about this paper is that they can use regularities, but I like that they put the images and text in the same embedding space by increasing similarity of positive samples and reducing similarity of negative ones - neat!

Tags

Paper Aliases

No aliases