Papperoni

2008

Deep Learning via Semi-Supervised Embedding

Jason Weston, F. Ratle, Ronan Collobert

Open PDF Google Scholar

citations

Cite Score

42

AI summary

This paper introduces a method for deep learning via semi-supervised embedding, where unsupervised techniques act as regularizers in deep architectures, yielding competitive error rates on datasets such as MNIST and COIL100.

Main Contributions

Introduces a novel semi-supervised learning method to improve the generalization ability of neural networks by utilizing unlabeled data.
Proposes to add a semi-supervised regularizer to deep architectures in one of three different modes.
Demonstrates that deep embedding is very easy to optimize by gradient descent, enabling efficient application to any kind of neural network architecture.
Shows that the approach can be cheaper to do gradient updates compared to a reconstruction-based loss function.
Achieves competitive results on several datasets, including MNIST and COIL100, showcasing the effectiveness of the proposed embedding method.

Abstract

We show how nonlinear embedding algorithms popular for use with "shallow" semi-supervised learning techniques such as kernel methods can be easily applied to deep multi-layer architectures, either as a regularizer at the output layer, or on each layer of the architecture. This trick provides a simple alternative to existing approaches to deep learning whilst yielding competitive error rates compared to those methods, and existing shallow semi-supervised techniques.

Citation Graph

Loading graph...

References [30]

Sort:

Filter:

[1]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[2]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[3]Multitask Learning

Rich Caruana - 1997

13 papers in library cite

I expected waaaaaay more from this paper. The idea is sooooo simple and the results are underwhelming. Also, 30 pages for something that could be said in 10. The writing style is a bit boring. TBH it seems like it's just a re-writing of Caruana's PhD thesis.

[4]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[5]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[6]Unsupervised Learning of Invariant Feature Hierarchies With Applications to Object Recognition

Marc'aurelio Ranzato, F. Huang, Y. Boureau, Yann Lecun - 2007

8 papers in library cite

I thought that I had learned pretty much everything before 2015, but this paper proved me wrong. Amazing read and very nice methodology. Very simple as well!

[7]Deep Learning via Semi-Supervised Embedding

Jason Weston, F. Ratle, Ronan Collobert - 2008

10 papers in library cite

It's a good paper and nice idea, but seems overly complicated and I don't think it's very used... (PS: this was republished in 2012)

[8]Statistical Learning Theory

V. N. Vapnik - 1998

10 papers in library cite

Book, but there's a 12 page overview

[9]A Global Geometric Framework for Nonlinear Dimensionality Reduction

J. Tenenbaum, V. D. Silva, John Langford - 2000

7 papers in library cite

[10]Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure

Ruslan Salakhutdinov, Geoffrey Hinton - 2007

5 papers in library cite

[11]Semi-Supervised Learning

O. Chapelle, B. Scholkopf, A. Zien - 2006

5 papers in library cite

[12]Columbia Object Image Library: Coil

S. Nene, S. Nayar, H. Murase - 1996

4 papers in library cite

[13]Deep Learning From Temporal Coherence in Video

H. Mobahi, Ronan Collobert, Jason Weston - 2009

3 papers in library cite

[14]Shallow Semantic Parsing Using Support Vector Machines

S. Pradhan, W. Ward, K. Hacioglu, J. Martin, Dan Jurafsky - 2004

3 papers in library cite

[15]Cluster Kernels for Semi-Supervised Learning

O. Chapelle, Jason Weston, B. Scholkopf - 2003

2 papers in library cite

[16]Fast Semantic Extraction Using a Novel Neural Network Architecture

Ronan Collobert, Jason Weston - 2007

2 papers in library cite

[17]Laplacian Eigenmaps for Dimensionality Reduction and Data Representation

M. Belkin, P. Niyogi - 2003

2 papers in library cite

[18]On a Connection Between Kernel PCA and Metric Multidimensional Scaling

C. Williams - 2001

2 papers in library cite

[19]Signature Verification Using a Siamese Time Delay Neural Network

J. Bromley, J. W. Bentz, Leon Bottou, I. Guyon, Yann Lecun, C. Moore, E. Sackinger, R. Shah - 1993

2 papers in library cite

[20]View-Based 3d Object Recognition With Support Vector Machines

D. Roobaert, M. M. V. Hulle - 1999

2 papers in library cite

[21]A Spin-Glass Markov Random Field for 3-D Object Recognition

B. Caputo, J. Hornegger, D. Paulus, H. Niemann - 2002

1 paper in library cites

[22]Beyond the Point Cloud: From Transductive to Semi-Supervised Learning

V. Sindhwani, P. Niyogi, M. Belkin - 2005

1 paper in library cites

[23]Dimensionality Reduction by Learning an Invariant Mapping

Raia Hadsell, S. Chopra, Yann Lecun - 2006

1 paper in library cites

[24]Large Scale Manifold Transduction

M. Karlen, Jason Weston, A. Erkan, Ronan Collobert - 2008

1 paper in library cites

[25]Large Scale Transductive SVMS

Ronan Collobert, F. Sinz, Jason Weston, Leon Bottou - 2006

1 paper in library cites

[26]Learning From Labeled and Unlabeled Data With Label Propagation

X. Zhu, Zoubin Ghahramani - 2002

1 paper in library cites

[27]Learning Optimized Features for Hierarchical Models of Invariant Recognition

H. Wersing, E. Korner - 2003

1 paper in library cites

[28]Manifold Regularization: A Geometric Framework for Learning From Labeled and Unlabeled Examples

M. Belkin, P. Niyogi, V. Sindhwani - 2006

1 paper in library cites

[29]Multidimensional Scaling by Optimizing Goodness of Fit to a Non-Metric Hypothesis

J. Kruskal - 1964

1 paper in library cites

[30]Semi-Supervised Classification by Low Density Separation

O. Chapelle, A. Zien - 2005

1 paper in library cites

Cited by

10

papers in your library

Cites

8

papers in your library

Read

on July 2, 2025

It's a good paper and nice idea, but seems overly complicated and I don't think it's very used... (PS: this was republished in 2012)

Tags

Paper Aliases

No aliases