Papperoni

2013

Representation Learning: A Review and New Perspectives

Yoshua Bengio, Aaron Courville, A. A. Vincent, P. Pascal

Open PDF Google Scholar

citations

Cite Score

91

AI summary

This paper reviews recent advancements in unsupervised feature learning and deep learning, focusing on probabilistic models, auto-encoders, and manifold learning, addressing questions about objectives for learning good representations and their geometrical connections.

Main Contributions

Reviews recent work in unsupervised feature learning and deep learning.
Covers advances in probabilistic models, auto-encoders, manifold learning, and deep networks.
Discusses the concept of 'good representations' and appropriate training objectives.
Explores the geometrical connections between representation learning, density estimation, and manifold learning.
Highlights the rapid increase in scientific activity and empirical successes of representation learning.

Abstract

The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although specific domain knowledge can be used to help design representations, learning with generic priors can also be used, and the quest for Al is motivating the design of more powerful representation-learning algorithms implementing such priors. This paper reviews recent work in the area of unsupervised feature learning and deep learning, covering advances in probabilistic models, auto-encoders, manifold learning, and deep networks. This motivates longer-term unanswered questions about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

Citation Graph

Loading graph...

References [215]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Visualizing Data Using t-SNE

Geoffrey Hinton - 2008

7 papers in library cite

Amazing. Simple. Impactful. Easy to understand. Masterpiece.

[3]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[4]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[5]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[6]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[7]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[8]Random Search for Hyper-Parameter Optimization

James Bergstra, Yoshua Bengio - 2012

7 papers in library cite

It seems crazy that it was only in 2012 that they found out that random search was good! Still, kudos on them for noticing, and the paper is just so easy to follow and enjoyable!

[9]Deep Neural Networks for Acoustic Modeling in Speech Recognition

Geoffrey Hinton - 2012

21 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

[10]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[11]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[12]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[13]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[14]Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories

Svetlana Lazebnik, Cordelia Schmid, Jean Ponce - 2006

14 papers in library cite

It's a fun read, but in the end is just an application of the spatial pyramid matching kernel from the other paper.

[15]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[16]Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position

Kunihiko Fukushima - 1980

8 papers in library cite

Good read! First conv net? :)

[17]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[18]Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network With a Local Denoising Criterion

P. H. Vincent, Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre Antoine Manzagol - 2010

6 papers in library cite

This is basically a summary of everything that happened from 2006-2010, and also points some interesting things about DBNs! Very well explained as well.

[19]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[20]Emergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images

B. Olshausen, D. Field - 1996

5 papers in library cite

I can see the contribution, but overall the paper was uninteresting

[21]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[22]Curriculum Learning

Yoshua Bengio, J. Louradour, Ronan Collobert, Jason Weston - 2009

6 papers in library cite

Very nice paper that introduces curriculum learning. Possibly not too relevant, but good nonetheless.

[23]Multi-Column Deep Neural Networks for Image Classification

Dan C. Ciresan, Ueli Meier, Jürgen Schmidhuber - 2012

11 papers in library cite

Very nice paper! And I am impressed they used CNNs before Hinton's paper. It's a shame there are so few citations. They also propose max-pooling and actually give a good explanation about it.

[24]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[25]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[26]Why Does Unsupervised Pre-Training Help Deep Learning?

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre Antoine Manzagol, Pascal Vincent, Samy Bengio - 2010

12 papers in library cite

Good paper, easy to follow, and brings some light to the pre-training stuff (layer-by-layer). I just wish it wasn't so long. It's a chore.

[27]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[28]Deconvolutional Networks

Matthew D. Zeiler, Dilip Krishnan, Graham W. Taylor, Rob Fergus - 2010

3 papers in library cite

It's a very nice idea but I feel that there is a much more intuitive way of explaining the idea.

[29]Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

3 papers in library cite

Nothing surprising really, just domain adaptation using NNs.

[30]Self-Taught Learning: Transfer Learning From Unlabeled Data

Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, A. Ng - 2007

7 papers in library cite

It's nice to make the distinction of self-taught learning vs. semi-supervised learning. However, it doesn't talk about NNs. Overall, LeCun's CNN unsupervised learning was better.

[31]Autoencoders, Minimum Description Length, and Helmholtz Free Energy

Geoffrey E. Hinton, Richard S. Zemel - 1994

3 papers in library cite

VERY MATHY. Doesn't explain anything. Overall very hard to understand.

[32]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[33]Efficient Learning of Sparse Representations With an Energy-Based Model

Marc'aurelio Ranzato, C. Poultney, S. Chopra, Yann Lecun - 2006

20 papers in library cite

It's ok. Not really good, but alright.

[34]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[35]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

 Probably seminal, but a bit boring overall

[36]Deep Learning Made Easier by Linear Transformations in Perceptrons

Tapani Raiko, Harri Valpola, Yann Lecun - 2012

7 papers in library cite

Kudos for introducing shortcut connections (which would become important in the future), but to me it seems a bit mid.

[37]Deep Learning via Hessian-Free Optimization

James Martens - 2010

12 papers in library cite

This paper is surprisingly good! When I first read the Hessian-Free optimization part, I thought "ugh, this is going to be full of math", but in the end it was very very enjoyable. I think I just wouldn't give it a 5 because it doesn't seem to have had that much impact.

[38]Deep Learning via Semi-Supervised Embedding

Jason Weston, F. Ratle, Ronan Collobert - 2008

10 papers in library cite

It's a good paper and nice idea, but seems overly complicated and I don't think it's very used... (PS: this was republished in 2012)

[39]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[40]Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent - 2012

8 papers in library cite

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

[41]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

13 papers in library cite

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[42]Measuring Invariances in Deep Networks

I. Goodfellow, Quoc Le, A. Saxe, A. Ng - 2009

7 papers in library cite

Very nice concept and methodology, but the results in the end are underwhelming

[43]Large Scale Image Annotation: Learning to Rank With Joint Word-Image Embeddings

Jason Weston, Samy Bengio, Nicolas Usunier - 2010

3 papers in library cite

Very simple approach! Seems like an early word2vec. I like the joint embeddings as well. However, results are underwhelming.

[44]Products of Experts

Geoffrey E. Hinton - 1999

4 papers in library cite

Very interesting concept. This is what paved the way to RBMs and DBNs in the future.

[45]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

Early work proving that NNs can be good. But very uninteresting overall.

[46]Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition

Dan C. Ciresan, Ueli Meier, Luca M. Gambardella, Jürgen Schmidhuber - 2010

10 papers in library cite

It's short, simple and straight to the point (as the network in the paper). It's refreshing to read something less "academic" and more "in your face"

[47]Practical Recommendations for Gradient-Based Training of Deep Architectures

Yoshua Bengio - 2012

3 papers in library cite

[48]Deep Boltzmann Machines

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

9 papers in library cite

Not sure what's the difference vs. DBNs

[49]Information Processing in Dynamical Systems: Foundations of Harmony Theory

P. Smolensky - 1986

11 papers in library cite

88 pages; Introduced RBMs

[50]A Practical Guide to Training Restricted boltzmann Machines

Geoffrey E. Hinton - 2010

4 papers in library cite

[51]Connectionist Learning of Belief Networks

R. Neal - 1992

2 papers in library cite

43 pages - intro to belief networks?

[52]Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing

Antoine Bordes, Xavier Glorot, Jason Weston, Yoshua Bengio - 2012

2 papers in library cite

[53]Unsupervised and Transfer Learning Challenge: A Deep Learning Approach

G. Mesnil, Yann Dauphin, Xavier Glorot, S. Rifai, Yoshua Bengio, I. Goodfellow, E. Lavoie, X. Muller, G. Desjardins, D. W. Farley, Pascal Vincent, Aaron Courville, J. Berkgstra - 2012

2 papers in library cite

Transfer learning with DNN

[54]Acoustic Modeling Using Deep Belief Networks

A. Mohamed, G. Dahl, Geoffrey Hinton - 2012

12 papers in library cite

[55]Sparse Feature Learning for Deep Belief Networks

Marc'aurelio Ranzato, Y. Boureau, Yann Lecun - 2008

12 papers in library cite

[56]Structured Output Layer neural Network Language Model

H. S. Le, I. Oparin, A. Allauzen, Jean Luc Gauvain, F. Yvon - 2011

7 papers in library cite

[57]Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation

Holger Schwenk, A. Rousseau, M. Attik - 2012

5 papers in library cite

[58]Practical Bayesian Optimization of Machine Learning Algorithms

J. Snoek, Hugo Larochelle, R. P. Adams - 2012

9 papers in library cite

[59]Hierarchical Models of Object Recognition in Cortex

M. Riesenhuber, T. Poggio - 1999

8 papers in library cite

[60]Unsupervised Learning of Distributions on Binary Vectors Using Two Layer Networks

Y. Freund, D. Haussler - 1992

8 papers in library cite

[61]A Global Geometric Framework for Nonlinear Dimensionality Reduction

J. Tenenbaum, V. D. Silva, John Langford - 2000

7 papers in library cite

[62]Exploring Strategies for Training Deep Neural Networks

Hugo Larochelle, Yoshua Bengio, J. Louradour, P. Lamblin - 2009

7 papers in library cite

40 pages

[63]Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models

M. Gutmann, A. Hyvarinen - 2010

7 papers in library cite

[64]On the Power of Small-Depth Threshold Circuits

J. Hastad, M. Goldmann - 1991

7 papers in library cite

[65]The Curse of Highly Variable Functions for Local Kernel Machines

Yoshua Bengio, O. Delalleau, N. L. Roux - 2006

7 papers in library cite

[66]Natural Gradient Works Efficiently in Learning

S. I. Amari - 1998

6 papers in library cite

[67]Object Recognition From Local Scale-Invariant Features

D. G. Lowe - 1999

6 papers in library cite

[68]Phone Recognition With the Mean-Covariance Restricted Boltzmann Machine

George E. Dahl, Marc'aurelio Ranzato, A. Mohamed, Geoffrey E. Hinton - 2010

6 papers in library cite

[69]Receptive Fields of Single Neurons in the Cat's Striate Cortex

D. H. Hubel, T. N. Wiesel - 1959

6 papers in library cite

[70]Justifying and Generalizing Contrastive Divergence

Yoshua Bengio, O. Delalleau - 2007

5 papers in library cite

[71]Learning Continuous Attractors in Recurrent Networks

S. H. Seung - 1998

5 papers in library cite

[72]Nonlinear Dimensionality Reduction by Locally Linear Embedding

S. T. Roweis, L. K. Saul - 2000

5 papers in library cite

[73]Restricted Boltzmann Machines for Collaborative Filtering

Ruslan Salakhutdinov, A. Mnih, Geoffrey E. Hinton - 2007

5 papers in library cite

[74]Semantic Hashing

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

[75]The Importance of Encoding Versus Training With Sparse Coding and Vector Quantization

A. Coates, Andrew Y. Ng - 2011

5 papers in library cite

[76]The Recurrent Temporal Restricted Boltzmann Machine

Ilya Sutskever, Geoffrey Hinton, G. Taylor - 2008

5 papers in library cite

[77]A Connection Between Score Matching and Denoising Autoencoders

Pascal Vincent - 2011

4 papers in library cite

[78]Almost Optimal Lower Bounds for Small Depth Circuits

J. Hastad - 1986

4 papers in library cite

[79]Auto-Association by Multilayer Perceptrons and Singular Value Decomposition

H. Bourlard, Y. Kamp - 1988

4 papers in library cite

[80]Best Practices for Convolutional Neural Networks

Patrice Y. Simard, D. Steinkraus, John C. Platt - 2003

4 papers in library cite

[81]Contracting Auto-Encoders: Explicit Invariance During Feature Extraction

S. Rifai, Pascal Vincent, X. Muller, Xavier Glorot, Yoshua Bengio - 2011

4 papers in library cite

[82]Conversational Speech Transcription Using Context-Dependent Deep Neural Networks

F. Seide, G. Li, D. Yu - 2011

4 papers in library cite

[83]Estimation of Non-Normalized Statistical Models Using Score Matching

A. Hyvarinen - 2005

4 papers in library cite

[84]ICA With Reconstruction Cost for Efficient Over-Complete Feature Learning

Quoc Le, A. Karpenko, J. Ngiam, A. Ng - 2011

4 papers in library cite

[85]Learning Hierarchical Invariant Spatio-Temporal Features for Action Recognition With Independent Subspace Analysis

Quoc Le, W. Zou, S. Y. Yeung, A. Ng - 2011

4 papers in library cite

[86]Learning Invariant Features Through Topographic Filter Maps

Koray Kavukcuoglu, Marc'aurelio Ranzato, Rob Fergus, Yann Lecun - 2009

4 papers in library cite

[87]On Optimization Methods for Deep Learning

Quoc V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, Andrew Y. Ng - 2011

4 papers in library cite

[88]Robust Object Recognition With Cortex-Like Mechanisms

T. Serre, Lior Wolf, S. Bileschi, M. Riesenhuber, T. Poggio - 2007

4 papers in library cite

[89]Tangent Prop-A Formalism for Specifying Selected Invariances in an Adaptive Neural Network

Patrice Simard, B. Victorri, Yann Lecun, John Denker - 1992

4 papers in library cite

[90]The "Independent Components" of Natural Scenes Are Edge Filters

A. Bell, T. Sejnowski - 1997

4 papers in library cite

[91]Tiled Convolutional Neural Networks

Quoc V. Le, J. Ngiam, Ziru Chen, D. Chia, P. W. Koh, Andrew Y. Ng - 2010

4 papers in library cite

[92]Training Restricted Boltzmann Machines Using Approximations to the Likelihood Gradient

T. Tieleman - 2008

4 papers in library cite

[93]A Generative Process for Sampling Contractive Auto-Encoders

S. Rifai, Yoshua Bengio, Yann Dauphin, Pascal Vincent - 2012

3 papers in library cite

[94]Algorithms for Hyper-Parameter Optimization

J. S. Bergstra, R. Bardenet, Yoshua Bengio, B. Kegl - 2011

3 papers in library cite

[95]Better Mixing via Deep Representations

Yoshua Bengio, G. Mesnil, Yann Dauphin, S. Rifai - 2013

3 papers in library cite

[96]Charting a Manifold

M. Brand - 2003

3 papers in library cite

[97]Convolutional Deep Belief Networks on cifar-10

Alex Krizhevsky - 2010

3 papers in library cite

[98]Convolutional Learning of Spatio-Temporal Features

Graham W. Taylor, Rob Fergus, Yann Lecun, C. Bregler - 2010

3 papers in library cite

[99]Convolutional Networks Can Learn to Generate Affinity Graphs for Image Segmentation

S. C. Turaga, J. F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, H. S. Seung - 2010

3 papers in library cite

[100]Decision Trees Do Not Generalize to New Variations

Yoshua Bengio, O. Delalleau, C. Simard - 2007

3 papers in library cite

[101]Deep Learning From Temporal Coherence in Video

H. Mobahi, Ronan Collobert, Jason Weston - 2009

3 papers in library cite

[102]Efficient Pattern Recognition Using a New Transformation Distance

Patrice Simard, Yann Lecun, John Denker - 1993

3 papers in library cite

[103]Emergence of Complex-Like Cells in a Temporal Product Network With Local Receptive Fields

K. Gregor, Yann Lecun - 2010

3 papers in library cite

[104]Factored 3-Way Restricted Boltzmann machines for Modeling Natural Images

Alex Krizhevsky, Geoffrey E. Hinton - 2010

3 papers in library cite

[105]Factored Conditional Restricted boltzmann Machines for Modeling Motion Style

Graham W. Taylor, Geoffrey E. Hinton - 2009

3 papers in library cite

[106]Fast Inference in Sparse Coding Algorithms With Applications to Object Recognition

Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2008

3 papers in library cite

[107]Learning Convolutional Feature Hierarchies for Visual Recognition

Koray Kavukcuoglu - 2010

3 papers in library cite

[108]Learning Fast Approximations of Sparse Coding

K. Gregor, Yann Lecun - 2010

3 papers in library cite

[109]Natural Image Denoising With Convolutional Networks

V. Jain, S. Seung - 2008

3 papers in library cite

[110]Nonlinear Component Analysis as a Kernel Eigenvalue Problem

B. Scholkopf, A. Smola, Klaus Robert Muller - 1998

3 papers in library cite

[111]On Contrastive Divergence Learning

M. A. C. Perpinan, Geoffrey E. Hinton - 2005

3 papers in library cite

[112]On the Convergence of Markovian Stochastic Algorithms With Rapidly Decreasing Ergodicity Rates

L. Younes - 1999

3 papers in library cite

[113]Slow Feature Analysis Yields a Rich Repertoire of Complex Cell Properties

P. Berkes, L. Wiskott - 2005

3 papers in library cite

[114]Training Recurrent Neural Networks

Ilya Sutskever - 2012

3 papers in library cite

[115]A Spike and slab Restricted Boltzmann Machine

Aaron Courville, James Bergstra, Yoshua Bengio - 2011

2 papers in library cite

[116]A Theoretical Analysis of Feature Pooling in Vision Algorithms

Y. Boureau, Jean Ponce, Yann Lecun - 2010

2 papers in library cite

[117]Classification Using Discriminative Restricted Boltzmann Machines

Hugo Larochelle, Yoshua Bengio - 2008

2 papers in library cite

[118]Deep Learning of Representations for Unsupervised and Transfer Learning

Yoshua Bengio - 2011

2 papers in library cite

[119]Document Image Defect Models

H. Baird - 1990

2 papers in library cite

[120]Empirical Evaluation of Convolutional RBMs for Vision

G. Desjardins, Yoshua Bengio - 2008

2 papers in library cite

[121]Evaluating Probabilities Under High-Dimensional Latent Variable Models

I. Murray, Ruslan Salakhutdinov - 2009

2 papers in library cite

[122]Feature Engineering in Context-Dependent Deep Neural Networks for Conversational Speech Transcription

F. Seide, G. Li, X. Chen, D. Yu - 2011

2 papers in library cite

[123]Higher Order Contractive Auto-Encoder

S. Rifai, G. Mesnil, Pascal Vincent, X. Muller, Yoshua Bengio, Yann Dauphin, Xavier Glorot - 2011

2 papers in library cite

[124]How Does the Brain Solve Visual Object Recognition?

J. J. Dicarlo, D. Zoccolan, N. C. Rust - 2012

2 papers in library cite

[125]Improved Local Coordinate Coding Using Local Tangents

K. Yu, Tong Zhang - 2010

2 papers in library cite

[126]Laplacian Eigenmaps for Dimensionality Reduction and Data Representation

M. Belkin, P. Niyogi - 2003

2 papers in library cite

[127]Learning Sparse Topographic Representations With Products of Student-T Distributions

M. Welling, Geoffrey Hinton, S. Osindero - 2003

2 papers in library cite

[128]Marginalized Denoising Autoencoders for Domain Adaptation

Mark Chen, Zhiwei Xu, K. Weinberger, F. Sha - 2012

2 papers in library cite

[129]Modeling Pixel Means and Covariances Using Factorized Third-Order Boltzmann machines

Marc'aurelio Ranzato, Geoffrey Hinton - 2010

2 papers in library cite

[130]Multimodal Learning With Deep Boltzmann Machines

N. Srivastava, Ruslan Salakhutdinov - 2012

2 papers in library cite

[131]Neural Net Language Models

Yoshua Bengio - 2008

2 papers in library cite

[132]Non-Local Manifold Parzen Windows

Yoshua Bengio, Hugo Larochelle, Pascal Vincent - 2006

2 papers in library cite

[133]Non-Local Manifold Tangent Learning

Yoshua Bengio, M. Monperrus - 2005

2 papers in library cite

[134]Nonlinear Learning Using Local Coordinate Coding

K. Yu, Tong Zhang, Y. Gong - 2009

2 papers in library cite

[135]On Deep Generative Models With Applications to Recognition

Marc'aurelio Ranzato, J. Susskind, V. Mnih, Geoffrey Hinton - 2011

2 papers in library cite

[136]Quickly Generating Representative Samples From an RBM-derived Process

O. Breuleux, Yoshua Bengio, Pascal Vincent - 2011

2 papers in library cite

[137]Regularized Estimation of Image Statistics by Score Matching

D. Kingma, Yann Lecun - 2010

2 papers in library cite

[138]Slow Feature Analysis: Unsupervised Learning of Invariances

L. Wiskott, T. Sejnowski - 2002

2 papers in library cite

[139]Statistical Analysis of Non-Lattice Data

J. Besag - 1975

2 papers in library cite

[140]Stochastic Neighbor Embedding

Geoffrey E. Hinton, S. T. Roweis - 2002

2 papers in library cite

[141]The Manifold Tangent classifier

S. Rifai, Yann Dauphin, Pascal Vincent, Yoshua Bengio, X. Muller - 2011

2 papers in library cite

[142]Transforming Auto-Encoders

Geoffrey E. Hinton, Alex Krizhevsky, S. D. Wang - 2011

2 papers in library cite

[143]Unsupervised Models of Images by Spike-and-Slab RBMs

Aaron Courville, James Bergstra, Yoshua Bengio - 2011

2 papers in library cite

[144]Using Fast Weights to Improve Persistent Contrastive Divergence

T. Tieleman, Geoffrey E. Hinton - 2009

2 papers in library cite

[145]What Regularized Auto-Encoders Learn From the Data Generating Distribution

G. Alain, Yoshua Bengio - 2013

2 papers in library cite

[146]When Does a Mixture of Products Contain a Product of Mixtures?

G. Montufar, J. Morton - 2014

2 papers in library cite

[147]A Connectionist Approach to Speech Recognition

Yoshua Bengio - 1993

1 paper in library cites

[148]A Self-Organizing Neural Network That Discovers Surfaces in Random-Dot Stereograms

S. Becker, Geoffrey Hinton - 1992

1 paper in library cites

[149]Ask the Locals: Multi-Way Local Pooling for Image Recognition

Y. Boureau, N. L. Roux, F. Bach, Jean Ponce, Yann Lecun - 2011

1 paper in library cites

[150]Asymptotic Efficiency of Deterministic Estimators for Discrete Energy-Based Models: Ratio Matching and Pseudolikelihood

B. Marlin, N. D. Freitas - 2011

1 paper in library cites

[151]Binary Coding of Speech Spectrograms Using a Deep Auto-Encoder

L. Deng, M. Seltzer, D. Yu, Alex Acero, A. Mohamed, Geoffrey Hinton - 2010

1 paper in library cites

[152]Blind Separation of Sources, Part I: An Adaptive Algorithm Based on Neuromimetic Architecture

C. Jutten, J. Herault - 1991

1 paper in library cites

[153]Boosted Backpropagation Learning for Training Deep Modular Networks

A. Grubb, J. A. D. Bagnell - 2010

1 paper in library cites

[154]Classification With Scattering Operators

J. Bruna, S. Mallat - 2011

1 paper in library cites

[155]Deep Coding Network

Yutong Lin, Z. Tong, S. Zhu, K. Yu - 2010

1 paper in library cites

[156]Differentiable Sparse Coding

J. A. Bagnell, D. M. Bradley - 2009

1 paper in library cites

[157]Echo state Network

Herbert Jaeger - 2007

1 paper in library cites

[158]Efficient Learning of Deep Boltzmann Machines

Ruslan Salakhutdinov, Hugo Larochelle - 2010

1 paper in library cites

[159]EM Algorithms for PCA and Sensible PCA

S. Roweis - 1997

1 paper in library cites

[160]Emergence of Phase and Shift Invariant Features by Decomposition of Natural Images Into Independent Feature Subspaces

A. Hyvarinen, P. Hoyer - 2000

1 paper in library cites

[161]Empirical Risk Minimization of Graphical Model Parameters Given Approximate Inference, Decoding, and Model Structure

Veselin Stoyanov, A. Ropson, J. Eisner - 2011

1 paper in library cites

[162]Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines

Kyunghyun Cho, Tapani Raiko, A. Ilin - 2011

1 paper in library cites

[163]Generating More Realistic Images Using Gated MRF's

Marc'aurelio Ranzato, V. Mnih, Geoffrey Hinton - 2010

1 paper in library cites

[164]Generative Versus Discriminative Training of RBMs for Classification of fMRI Images

T. Schmah, Geoffrey E. Hinton, R. Zemel, S. L. Small, S. Strother - 2009

1 paper in library cites

[165]Group Invariant Scattering

S. Mallat - 2012

1 paper in library cites

[166]Herding Dynamic Weights for Partially Observed Random Field Models

M. Welling - 2009

1 paper in library cites

[167]Hessian Eigenmaps: New Locally Linear Embedding Techniques for High-Dimensional Data

D. L. Donoho, C. Grimes - 2003

1 paper in library cites

[168]How Are Complex Cell Properties Adapted to the Statistics of Natural Stimuli?

K. P. Kording, C. Kayser, W. Einhauser, P. Konig - 2004

1 paper in library cites

[169]Implicit Density Estimation by Local Moment Matching to Sample From Auto-Encoders

Yoshua Bengio, G. Alain, S. Rifai - 2012

1 paper in library cites

[170]Independent Component Analysis

A. Hyvarinen, J. Karhunen, E. Oja - 2001

1 paper in library cites

[171]Inductive Principles for Restricted Boltzmann Machine Learning

B. Marlin, K. Swersky, Berlin Chen, N. D. Freitas - 2010

1 paper in library cites

[172]Inductive Principles for Restricted Boltzmann Machine Learning

B. Marlin, K. Swersky, Berlin Chen, N. D. Freitas - 2010

1 paper in library cites

[173]Learning a Parametric Embedding by Preserving Local Structure

Laurens Van Der Maaten - 2009

1 paper in library cites

[174]Learning Approximate Inference Policies for Fast Prediction

J. Eisner - 2012

1 paper in library cites

[175]Learning Deep Boltzmann Machines Using Adaptive McMc

Ruslan Salakhutdinov - 2010

1 paper in library cites

[176]Learning Deep Energy Models

J. Ngiam, Ziru Chen, P. Koh, A. Ng - 2011

1 paper in library cites

[177]Learning Image Representations From the Pixel Level via Hierarchical Sparse Coding

K. Yu, Yutong Lin, J. Lafferty - 2011

1 paper in library cites

[178]Learning in Markov Random Fields Using Tempered Transitions

Ruslan Salakhutdinov - 2010

1 paper in library cites

[179]Learning to Represent Spatial Transformations With Factored Higher-Order Boltzmann Machines

R. Memisevic, Geoffrey E. Hinton - 2010

1 paper in library cites

[180]Learning Transformational Invariants From Natural Movies

C. Cadieu, B. Olshausen - 2009

1 paper in library cites

[181]Manifold Parzen Windows

Pascal Vincent, Yoshua Bengio - 2003

1 paper in library cites

[182]Multiple Texture Boltzmann Machines

J. J. Kivinen, Christopher K. I. Williams - 2012

1 paper in library cites

[183]Natural Gradient Revisited

Razvan Pascanu, Yoshua Bengio - 2013

1 paper in library cites

[184]Natural Image Statistics: A Probabilistic Approach to Early Computational Vision

A. Hyvarinen, J. Hurri, P. O. Hoyer - 2009

1 paper in library cites

[185]Neural Networks: Tricks of the Trade

G. Orr, Klaus Robert Muller - 1998

1 paper in library cites

[186]On Score Matching for Energy Based Models: Generalizing Autoencoders and Simplifying Deep Learning

K. Swersky, Marc'aurelio Ranzato, D. Buchman, B. Marlin, N. D. Freitas - 2011

1 paper in library cites

[187]On the Convergence Properties of Contrastive Divergence

Ilya Sutskever, T. Tieleman - 2010

1 paper in library cites

[188]On the Expressive Power of Deep Architectures

Yoshua Bengio, O. Delalleau - 2011

1 paper in library cites

[189]On Tracking the Partition Function

G. Desjardins, Aaron Courville, Yoshua Bengio - 2011

1 paper in library cites

[190]On Training Deep Boltzmann Machines

G. Desjardins, Aaron Courville, Yoshua Bengio - 2012

1 paper in library cites

[191]Optimal Approximation of Signal Priors

A. Hyvarinen - 2008

1 paper in library cites

[192]Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering

Yoshua Bengio, J. F. Paiement, Pascal Vincent, O. Delalleau, N. L. Roux, M. Ouimet - 2004

1 paper in library cites

[193]Parallel Tempering Is Efficient for Learning Restricted Boltzmann Machines

Kyunghyun Cho, Tapani Raiko, A. Ilin - 2010

1 paper in library cites

[194]Probabilistic Inference Using Markov chain Monte-Carlo Methods

Radford M. Neal - 1993

1 paper in library cites

[195]Probabilistic Principal Components Analysis

M. E. Tipping, C. M. Bishop - 1999

1 paper in library cites

[196]Réseaux de Neurones À Relaxation Entraînés par critère d'autoencodeur Débruitant

F. Savard - 2011

1 paper in library cites

[197]Selecting Receptive Fields in Deep Networks

A. Coates, Andrew Y. Ng - 2011

1 paper in library cites

[198]Sequential Labeling Using Deep-Structured Conditional Random Fields

D. Yu, Shijie Wang, L. Deng - 2010

1 paper in library cites

[199]Shift-Invariant Sparse Coding for Audio Classification

R. Grosse, Rajat Raina, H. Kwong, Andrew Y. Ng - 2007

1 paper in library cites

[200]Should Penalized Least Squares Regression Be Interpreted as Maximum a Posteriori Estimation?

R. Gribonval - 2011

1 paper in library cites

[201]Slow, Decorrelated Features for Pretraining Complex Cell-Like Networks

James Bergstra, Yoshua Bengio - 2009

1 paper in library cites

[202]Some Extensions of Score Matching

A. Hyvarinen - 2007

1 paper in library cites

[203]Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery

I. Goodfellow, Aaron Courville, Yoshua Bengio - 2011

1 paper in library cites

[204]Structured Sparse Coding via Lateral Inhibition

K. Gregor, A. Szlam, Yann Lecun - 2011

1 paper in library cites

[205]Structured Sparsity Through Convex Optimization

F. Bach, R. Jenatton, Julien Mairal, G. Obozinski - 2011

1 paper in library cites

[206]Structured Variable Selection With Sparsity-Inducing Norms

R. Jenatton, J. Y. Audibert, F. Bach - 2009

1 paper in library cites

[207]Tempered Markov Chain Monte Carlo for Training of Restricted Boltzmann Machine

G. Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, O. Delalleau - 2010

1 paper in library cites

[208]Temporal Coherence, Natural Image Sequences, and the Visual Cortex

J. Hurri, A. Hyvarinen - 2003

1 paper in library cites

[209]Temporal Pooling and Multiscale Learning for Automatic Annotation and Ranking of Music Audio

P. Hamel, S. Lemieux, Yoshua Bengio, D. Eck - 2011

1 paper in library cites

[210]The Convergence of Contrastive Divergences

A. L. Yuille - 2005

1 paper in library cites

[211]Topographic Independent Component Analysis

A. Hyvarinen, P. O. Hoyer, M. Inki - 2001

1 paper in library cites

[212]Understanding Representations Learned in Deep Architectures

Dumitru Erhan, Aaron Courville, Yoshua Bengio - 2010

1 paper in library cites

[213]Unsupervised Learning of Image Manifolds by semidefinite Programming

K. Q. Weinberger, L. K. Saul - 2004

1 paper in library cites

[214]Unsupervised Learning of Sparse Features for Scalable Audio Classification

M. Henaff, K. Jarrett, Koray Kavukcuoglu, Yann Lecun - 2011

1 paper in library cites

[215]Unsupervised Learning of Visual Invariance With Temporal Coherence

W. Y. Zou, Andrew Y. Ng, K. Yu - 2011

1 paper in library cites

Cited by

4

papers in your library

Cites

55

papers in your library

Read

on February 21, 2026

Good in spirit but is too long and covers a lot of things that are not relevant anymore.

Tags

Paper Aliases

No aliases