Papperoni

2013

Pylearn2: A Machine Learning Research Library

Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

17

AI summary

This paper introduces Pylearn2, a machine learning research library designed for flexibility and extensibility, which facilitates research projects with new use cases; it uses Theano to provide a language for describing expressions and it is implemented in Python.

Main Contributions

Pylearn2 is a machine learning research library designed for flexibility and extensibility.
Pylearn2 is built from reusable parts that can be used in many combinations or independently.
Pylearn2 avoids over-planning by ensuring designs are modular and easy to extend.
Pylearn2 provides a domain-specific language that provides a compact way of specifying all hyperparameters for an experiment.
Pylearn2 consists of several components that can be combined to form complete learning algorithms.

Abstract

Pylearn2 is a machine learning research library. This does not just mean that it is a collection of machine learning algorithms that share a common API; it means that it has been designed for flexibility and extensibility in order to facilitate research projects that involve new or unusual use cases. In this paper we give a brief history of the library, an overview of its basic philosophy, a summary of the library's architecture, and a description of how the Pylearn2 community functions socially.

Citation Graph

Loading graph...

References [56]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[3]Learning Multiple Layers of Features From Tiny Images

Alex Krizhevsky - 2009

27 papers in library cite

It's alright. It mainly focuses on RBMs and their features and the actual part that describes the dataset is like 1 page. However, it's maybe the best intuitive description of an RBM I have seen. Other than that, it reads very much like an undergraduate thesis.

[4]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[5]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[6]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[7]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[8]Reading Digits in Natural Images With Unsupervised Feature Learning

Y. Netzer, Tianle Wang, A. Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng - 2011

8 papers in library cite

It's a bit meh. It's also 2011 so maybe it was impressive, but I think the main contribution is the dataset.

[9]Training Products of Experts by Minimizing Contrastive Divergence

Geoffrey Hinton - 2002

23 papers in library cite

Good read, but I think I need to revisit it after I understand RBMs better.

[10]Large Scale Distributed Deep Networks

Jeffrey Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Quoc V. Le, Mark Z. Mao, Marc'aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Andrew Y. Ng - 2012

16 papers in library cite

Good paper, nice algorithm. Nothing too crazy, but I understand the impact. I think the work to create the system was larger than the algorithm itself.

[11]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[12]Torch7: A Matlab-Like Environment for Machine Learning

Clement Farabet - 2011

5 papers in library cite

Very nice paper! I am impressed that they used Lua, and the API sounds very simple. Plus, it sounds like they didn't even try that hard to get good performance (vs. Theano which was very tryhard)

[13]Learning Methods for Generic Object Recognition With Invariance to Pose and Lighting

Yann Lecun, Fu Jie Huang, Leon Bottou - 2004

18 papers in library cite

Good paper, nice methodology for creating different images. However, I think that this was not too impactful... I don't see this being used a lot.

[14]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[15]An Empirical Evaluation of Deep Architectures on Problems With Many Factors of Variation

Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio - 2007

13 papers in library cite

Good paper showing promising results for Deep Learning. Nothing amazing but good nonetheless

[16]Convolutional neural Networks Applied to House Numbers Digit Classification

P. Sermanet, S. Chintala, Yann Lecun - 2012

6 papers in library cite

This reads like an undergrad's research paper or a recent master's paper. Only applying stuff to a new dataset.

[17]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[18]Support Vector Networks

C. Cortes, V. Vapnik - 1995

3 papers in library cite

Tons of citations, SVMs?

[19]Deep Boltzmann Machines

Ruslan Salakhutdinov, Geoffrey E. Hinton - 2009

9 papers in library cite

Not sure what's the difference vs. DBNs

[20]Information Processing in Dynamical Systems: Foundations of Harmony Theory

P. Smolensky - 1986

11 papers in library cite

88 pages; Introduced RBMs

[21]Exponential Family Harmoniums With an Application to Information Retrieval

M. Welling, M. R. Zvi, Geoffrey Hinton - 2005

8 papers in library cite

[22]An Analysis of Single-Layer Networks in Unsupervised Feature Learning

A. Coates, A. Ng, Honglak Lee - 2011

7 papers in library cite

[23]Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models

M. Gutmann, A. Hyvarinen - 2010

7 papers in library cite

[24]A Connection Between Score Matching and Denoising Autoencoders

Pascal Vincent - 2011

4 papers in library cite

[25]Acceleration of Stochastic Approximation by Averaging

B. T. Polyak, A. B. Juditsky - 1992

4 papers in library cite

[26]Auto-Association by Multilayer Perceptrons and Singular Value Decomposition

H. Bourlard, Y. Kamp - 1988

4 papers in library cite

[27]Contracting Auto-Encoders: Explicit Invariance During Feature Extraction

S. Rifai, Pascal Vincent, X. Muller, Xavier Glorot, Yoshua Bengio - 2011

4 papers in library cite

[28]Estimation of Non-Normalized Statistical Models Using Score Matching

A. Hyvarinen - 2005

4 papers in library cite

[29]The "Independent Components" of Natural Scenes Are Edge Filters

A. Bell, T. Sejnowski - 1997

4 papers in library cite

[30]Training Restricted Boltzmann Machines Using Approximations to the Likelihood Gradient

T. Tieleman - 2008

4 papers in library cite

[31]Convolutional Networks for Images, Speech, and Time-Series

Yann Lecun, Yoshua Bengio - 1995

3 papers in library cite

[32]On the Convergence of Markovian Stochastic Algorithms With Rapidly Decreasing Ergodicity Rates

L. Younes - 1999

3 papers in library cite

[33]Python for Scientific Computing

T. E. Oliphant - 2007

3 papers in library cite

[34]Rank, Trace-Norm and Max-Norm

N. Srebro, A. Shraibman - 2005

3 papers in library cite

[35]A Spike and slab Restricted Boltzmann Machine

Aaron Courville, James Bergstra, Yoshua Bengio - 2011

2 papers in library cite

[36]Higher Order Contractive Auto-Encoder

S. Rifai, G. Mesnil, Pascal Vincent, X. Muller, Yoshua Bengio, Yann Dauphin, Xavier Glorot - 2011

2 papers in library cite

[37]Nonlinear Learning Using Local Coordinate Coding

K. Yu, Tong Zhang, Y. Gong - 2009

2 papers in library cite

[38]The Toronto Face Dataset

J. Susskind, A. Anderson, Geoffrey E. Hinton - 2010

2 papers in library cite

[39]Unsupervised Models of Images by Spike-and-Slab RBMs

Aaron Courville, James Bergstra, Yoshua Bengio - 2011

2 papers in library cite

[40]Challenges in Representation Learning: A Report on Three Machine Learning Contests

I. Goodfellow, Dumitru Erhan, P. Carrier, Aaron Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D. L. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, Xinpeng Wang, D. Athanasakis, J. S. Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, James Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, Yoshua Bengio - 2013

1 paper in library cites

[41]Computer-Intensive Methods in Statistics

P. Diaconis, B. Efron - 1983

1 paper in library cites

Missing author list

2013

1 paper in library cites

[43]Invariant Object Recognition With Slow Feature Analysis

M. Franzius, N. Wilbert, L. Wiskott - 2008

1 paper in library cites

[44]Large-Scale Learning of Embeddings With Reconstruction Sampling

Yann Dauphin, Xavier Glorot, Yoshua Bengio - 2011

1 paper in library cites

[45]NIPS Workshop on Challenges in Learning Hierarchical Models: Transfer Learning and Optimization

Quoc V. Le, Marc'aurelio Ranzato, Ruslan Salakhutdinov, A. Ng, J. Tenenbaum - 2011

1 paper in library cites

[46]Note sur la convergence de Méthodes De Directions Conjuguées

E. Polak, G. Ribiere - 1969

1 paper in library cites

[47]On Lines and Planes of Closest Fit to Systems of Points in Space

K. Pearson - 1901

1 paper in library cites

[48]Scaling Up Spike-and-Slab Models for Ansupervised Feature Learning

I. Goodfellow, Aaron Courville, Yoshua Bengio - 2013

1 paper in library cites

[49]Scikit-Learn: Machine Learning in Python

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay - 2011

1 paper in library cites

[50]Sur la division des corps matériels en Parties

H. Steinhaus - 1957

1 paper in library cites

[51]The Collected Papers of Peter J.W. Debye

P. Debye - 1954

1 paper in library cites

[52]The OpenCV Library

G. Bradski - 2000

1 paper in library cites

[53]The Stair Vision Library

S. Gould, O. Russakovsky, I. Goodfellow, P. Baumstarck, Andrew Y. Ng, D. Koller - 2010

1 paper in library cites

[54]The Use of Multiple Measurements in Taxonomic Problems

R. A. Fisher - 1936

1 paper in library cites

Missing author list

2013

1 paper in library cites

[56]Unsupervised and Transfer Learning Challenge

I. Guyon, G. Dror, V. Lemaire, G. Taylor, D. W. Aha - 2011

1 paper in library cites

Cited by

5

papers in your library

Cites

20

papers in your library

Read

on July 25, 2025

The paper itself is ok, but I think it had little impact. The idea of having an extensible and easy to use framework is nice (and is something that I also try to do in my projects), but seems like it didn't catch on and was only used by Bengio's lab.

Tags

Paper Aliases

No aliases