Papperoni

2015

DRAW: A Recurrent Neural Network for Image Generation

K. Gregor, Ivo Danihelka, Alex Graves, D. J. Rezende, Daan Wierstra

Open PDF Google Scholar

citations

Cite Score

60

AI summary

This paper introduces the Deep Recurrent Attentive Writer (DRAW) network, combining spatial attention and a variational auto-encoding framework for iterative image construction. It improves generative models on MNIST and generates realistic images on the Street View House Numbers dataset.

Main Contributions

Introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation.
Combines a novel spatial attention mechanism that mimics the foveation of the human eye.
Introduces a sequential variational auto-encoding framework that allows for the iterative construction of complex images.
Improves on the state of the art for generative models on MNIST.
Generates images that cannot be distinguished from real data with the naked eye when trained on the Street View House Numbers dataset.

Abstract

This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.

Citation Graph

Loading graph...

References [32]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[4]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[5]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[6]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[7]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[8]Reading Digits in Natural Images With Unsupervised Feature Learning

Y. Netzer, Tianle Wang, A. Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng - 2011

8 papers in library cite

It's a bit meh. It's also 2011 so maybe it was impressive, but I think the main contribution is the dataset.

[9]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[10]Recurrent Models of Visual Attention

V. Mnih, N. Heess, Alex Graves - 2014

5 papers in library cite

It's not as good as the other paper (DRAW), but it's a precursor and it's so nice how the model learns to pay attention. Also very nice to see RL in the mix, and see the possible usages in games and other things.

[11]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[12]Learning Generative Models With Visual Attention

Ruslan Salakhutdinov - 2014

2 papers in library cite

Waaaay to mathy and TBH I think there aren't a lot of good results. But also I don't have a ton of knowledge on generative models, so I may be biased.

[13]Deep Boltzmann Machines

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

9 papers in library cite

Not sure what's the difference vs. DBNs

[14]Multiple Object Recognition With Visual Attention

Jimmy Lei Ba, V. Mnih, Koray Kavukcuoglu - 2014

4 papers in library cite

Cited by DRAW

[15]Multi-Digit Number Recognition From Street View Imagery Using Deep Convolutional Neural Networks

Ian J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, V. Shet - 2014

4 papers in library cite

Goodfellow, street number recog

[16]On Learning Where to Look

Marc'aurelio Ranzato - 2014

3 papers in library cite

Learning where to look with glimpses

[17]The Neural Autoregressive Distribution Estimator

Hugo Larochelle, I. Murray - 2011

5 papers in library cite

[18]Learning to Combine Foveal Glimpses With a Third-Order Boltzmann Machine

Hugo Larochelle, Geoffrey E. Hinton - 2010

4 papers in library cite

[19]Learning Where to Attend With Deep Architectures for Image Tracking

M. Denil, L. Bazzani, Hugo Larochelle, N. D. Freitas - 2012

4 papers in library cite

[20]On the Quantitative Analysis of Deep Belief Networks

Ruslan Salakhutdinov, I. Murray - 2008

4 papers in library cite

[21]Auto-Encoding Variational Bayes

D. P. Kingma, M. Welling - 2013

3 papers in library cite

[22]Stochastic Backpropagation and Approximate Inference in Deep Generative Models

D. J. Rezende, S. Mohamed, Daan Wierstra - 2014

3 papers in library cite

[23]Evaluating Probabilities Under High-Dimensional Latent Variable Models

I. Murray, Ruslan Salakhutdinov - 2009

2 papers in library cite

[24]The Helmholtz Machine

Peter Dayan, Geoffrey Hinton, R. Neal, R. Zemel - 1995

2 papers in library cite

[25]A Deep and Tractable Density Estimator

B. Uria, I. Murray, Hugo Larochelle - 2014

1 paper in library cites

[26]A Neural Autoregressive Approach to Attention-Based Recognition

Y. Zheng, Richard S. Zemel, Y. Z. Zhang, Hugo Larochelle - 2014

1 paper in library cites

[27]Attention for Fine-Grained Categorization

P. Sermanet, Andrea Frome, A. Andrea, E. Real - 2014

1 paper in library cites

[28]Deep Autoregressive Networks

K. Gregor, Ivo Danihelka, A. Mnih, A. Andriy, C. Blundell, C. Charles, Daan Wierstra - 2014

1 paper in library cites

[29]Iterative Neural Autoregressive Distribution Estimator Nade-K

Tapani Raiko, Yiwei Li, Y. Yao, Kyunghyun Cho, K. Kyunghyun, Yoshua Bengio, Y. Yoshua - 2014

1 paper in library cites

[30]Markov chain monte carlo and variational Inference: Bridging the Gap

T. Salimans, D. P. Kingma, P. Diederik, M. Welling - 2014

1 paper in library cites

[31]Neural Variational Inference and Learning in Belief Networks

A. Mnih, K. Gregor - 2014

1 paper in library cites

[32]Optimizing Neural Networks That Generate Images

T. Tieleman - 2014

1 paper in library cites

Cited by

5

papers in your library

Cites

16

papers in your library

Read

on August 2, 2025

This is SO cool! So interesting that it was biologically inspired. First actual image generation that seems to work, also a jump from the RBM/Autoencoder stuff, and the incremental drawing is amazing!

Tags

Paper Aliases

No aliases