Papperoni

2007

An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio

citations

Cite Score

AI summary

This paper introduces deep belief networks (DBN-3) and stacked autoencoders (SAA-3) models and compares their performance with other algorithms on datasets with many factors of variation, such as MNIST variations and convex set recognition, finding that deep architecture models generally outperform shallow models but are sensitive to hyper-parameter selection.

Main Contributions

Introduces a suite of datasets that spans some of the territory between MNIST and NORB-starting with MNIST, and introducing multiple factors of variation such as rotation and background manipulations.
Demonstrates that deep architecture models show globally the best performance on the introduced datasets.
Shows that the improvement provided by deep architecture models is most notable for factors of variation related to background.
Provides empirical evidence that deep architecture models compare favorably to other state-of-the-art learning algorithms on learning problems with many factors of variation.
Analyzes the relationships between the performance of learning algorithms and certain properties of the problems considered.

Abstract

Recently, several learning algorithms relying on models with deep architectures have been proposed. Though they have demonstrated impressive performance, to date, they have only been evaluated on relatively simple problems such as digit recognition in a controlled environment, for which many machine learning algorithms already report reasonable results. Here, we present a series of experiments which indicate that these models show promise in solving harder learning problems that exhibit many factors of variation. These models are compared with well-established algorithms such as Support Vector Machines and single hidden-layer feed-forward neural networks.

Citation Graph

Loading graph...

References [12]

Sort:

Filter:

[1]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

Google Scholar

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[2]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

Google Scholar

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[3]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

Google Scholar

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[4]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Google Scholar

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[5]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Google Scholar

Good read, but I think I need to revisit it after I understand RBMs better.

[6]Scaling Learning Algorithms Towards AI

Yoshua Bengio, Yann Lecun - 2007

15 papers in library cite

Google Scholar

I should have read this sooner! Such a good explanation of why deep learning > other stuff! Also, better than Bengio's 2006 Learning Deep Archs for AI

[7]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

Google Scholar

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[8]To Recognize Shapes, First Learn to Generate Images

Geoffrey Hinton - 2006

5 papers in library cite

Google Scholar

Maybe the best explanation of deep belief nets and RBMs by Hinton.

[9]Exponential Family Harmoniums With an Application to Information Retrieval

M. Welling, M. R. Zvi, Geoffrey Hinton - 2005

8 papers in library cite

Google Scholar

[10]Training Invariant Support Vector Machines

D. Decoste, B. Scholkopf - 2002

6 papers in library cite

Google Scholar

[11]Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

Google Scholar

[12]LIBSVM: A Library for Support Vector Machines

C. C. Chang, C. J. Lin - 2001

4 papers in library cite

Google Scholar

Cited by

papers in your library

Cites

papers in your library

Read

on July 31, 2025

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless