Papperoni

2015

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler

Open PDF Google Scholar

citations

Cite Score

69

AI summary

This paper introduces a model that aligns books to movies to provide rich descriptive explanations for visual content, using a neural sentence embedding trained on the BookCorpus dataset and a video-text neural embedding trained on DVS descriptions of movie clips. It achieves good performance for movie/book alignment.

Main Contributions

Introduces a model that aligns books to movies to provide rich descriptive explanations for visual content.
Exploits a neural sentence embedding trained on the BookCorpus dataset.
Uses a video-text neural embedding trained on DVS descriptions of movie clips.
Proposes a context-aware CNN to combine information from multiple sources.
Introduces the MovieBook dataset with 11 movie/book pairs annotated with 2,070 shot-to-sentence correspondences.

Abstract

Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[4]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[5]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[6]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[7]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[8]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

Very cool idea. Simple yet very impactful!

[9]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[10]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[11]Show, Attend and Tell: Neural Image Caption Generation With Visual Attention

K. Xu, Jimmy Lei Ba, R. Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, R. Zemel, Yoshua Bengio - 2015

12 papers in library cite

It's a nice paper. I liked the soft attention way more than the hard one, and I am a bit mad that it wasn't the best lol And also it's the first paper I read about multimodality, but it seems that this was bustling at the time. Also results are kinda bad.

[12]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[13]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[14]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[15]Unifying Visual-Semantic Embeddings With Multimodal Neural Language Models

Richard S. Zemel - 2014

5 papers in library cite

I think what I like the most about this paper is that they can use regularities, but I like that they put the images and text in the same embedding space by increasing similarity of positive samples and reducing similarity of negative ones - neat!

[16]Deep Visual-Semantic Alignments for Generating Image Descriptions

A. Karpathy, Li Fei Fei - 2014

6 papers in library cite

Early multimodality

[17]Translating Videos to Natural Language Using Deep Recurrent Neural Networks

S. Venugopalan, Hu Xu, J. Donahue, M. Rohrbach, R. J. Mooney, K. Saenko - 2014

1 paper in library cites

Video to text

[18]A Dataset for Movie Description

A. Rohrbach, M. Rohrbach, N. Tandon, B. Schiele - 2015

1 paper in library cites

Dataset for movie description

[19]Explain Images With Multimodal Recurrent Neural Networks

J. Mao, Weixin Xu, Yining Yang, J. Wang, A. L. Yuille - 2014

3 papers in library cite

Baseline for later captioning, improved on initial multimodel paper

[20]Inferring the Why in Images

H. Pirsiavash, C. Vondrick, Antonio Torralba - 2014

1 paper in library cites

[21]Baby Talk: Understanding and Generating Simple Image Descriptions

G. Kulkarni, V. Premraj, S. Dhar, Shanda Li, Yejin Choi, A. C. Berg, T. L. Berg - 2011

4 papers in library cite

[22]Every Picture Tells a Story: Generating Sentences From Images

Ali Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, David Forsyth - 2010

3 papers in library cite

[23]Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers

Aman Gupta, L. S. Davis - 2008

2 papers in library cite

[24]"Hello! My Name Is... Buffy" - Automatic Naming of Characters in TV Video

Mark Everingham, Josef Sivic, Andrew Zisserman - 2006

1 paper in library cites

[25]"Who Are You?" Learning Person Specific Classifiers From Video

Josef Sivic, Mark Everingham, Andrew Zisserman - 2009

1 paper in library cites

[26]A Multi-World Approach to Question Answering About Real-World Scenes Based on Uncertain Input

M. Malinowski, M. Fritz - 2014

1 paper in library cites

[27]A Sentence Is Worth a Thousand Pixels

Sanja Fidler, Archit Sharma, R. Urtasun - 2013

1 paper in library cites

[28]Aligning Plot Synopses to Videos for Story-Based Retrieval

M. Tapaswi, M. Bauml, R. Stiefelhagen - 2015

1 paper in library cites

[29]Book2Movie: Aligning Video Scenes With Book Chapters

M. Tapaswi, M. Bauml, R. Stiefelhagen - 2015

1 paper in library cites

[30]Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks

Xiangning Lin, D. Parikh - 2015

1 paper in library cites

[31]Efficient Structured Prediction With Latent Variables for General Graphical Models

A. Schwing, T. Hazan, M. Pollefeys, R. Urtasun - 2012

1 paper in library cites

[32]Learning Deep Features for Scene Recognition Using Places Database

B. Zhou, A. Lapedriza, Jianxiong Xiao, Antonio Torralba, Aude Oliva - 2014

1 paper in library cites

[33]Linking People in Videos With "Their" Names Using Coreference Resolution

V. Ramanathan, Armand Joulin, Percy Liang, Li Fei Fei - 2014

1 paper in library cites

[34]Movie/Script: Alignment and Parsing of Video and Text Transcription

T. Cour, C. Jordan, E. Miltsakaki, B. Taskar - 2008

1 paper in library cites

[35]Subtitle-Free Movie to Script Alignment

P. Sankar, C. V. Jawahar, Andrew Zisserman - 2009

1 paper in library cites

[36]Video Event Understanding Using Natural Language Descriptions

V. Ramanathan, Percy Liang, Li Fei Fei - 2013

1 paper in library cites

[37]Visual Semantic Search: Retrieving Videos via Complex Textual Queries

D. Lin, Sanja Fidler, C. Kong, R. Urtasun - 2014

1 paper in library cites

[38]What Are You Talking About? Text-to-Image Coreference

C. Kong, D. Lin, Mohit Bansal, R. Urtasun, Sanja Fidler - 2014

1 paper in library cites

Cited by

18

papers in your library

Cites

20

papers in your library

Read

on August 5, 2025

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

Tags

Paper Aliases

No aliases