Papperoni

2015

Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le

Open PDF Google Scholar

citations

Cite Score

51

AI summary

This paper introduces two semi-supervised learning approaches for sequence learning with recurrent networks: predicting the next sequence element and sequence autoencoders. LSTMs pretrained with these approaches show improved stability and generalization, achieving strong performance on text classification tasks such as IMDB, DBpedia and 20 Newsgroups.

Main Contributions

Introduces a sequence autoencoder approach to semi-supervised learning for sequence learning with recurrent networks.
Demonstrates that pretraining LSTMs with sequence autoencoders improves stability and generalization.
Shows that LSTMs pretrained with sequence autoencoders achieve strong performance on text classification tasks.
Shows that using more unlabeled data from related tasks in the pretraining can improve the generalization of a subsequent supervised model.
Achieves state-of-the-art results on IMDB, DBpedia and 20 Newsgroups text classification datasets.

Abstract

We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural language processing. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a “pretraining” step for a later supervised sequence learning algorithm. In other words, the parameters obtained from the unsupervised step can be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after being pretrained with the two approaches are more stable and generalize better. With pretraining, we are able to train long short term memory recurrent networks up to a few hundred timesteps, thereby achieving strong performance in many text classification tasks, such as IMDB, DBpedia and 20 Newsgroups.

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[4]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[5]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[6]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[7]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[8]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[9]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[10]LSTM: A Search Space Odyssey

K. Greff, R. K. Srivastava, J. Koutn'ik, B. R. Steunebrink, Jürgen Schmidhuber - 2015

4 papers in library cite

Very good review on different architectural choices of LSTM -  actually brings some nice insights (as opposed to the other paper that compares GRU and LSTMs)

[11]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[12]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[13]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[14]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[15]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[16]A Neural Conversational Model

Oriol Vinyals, Quoc V. Le - 2015

7 papers in library cite

No new methodology, no measurements, no results. It should be like 4 pages long if they didn't fill in with the conversation logs.

[17]A Framework for Learning Predictive Structures From Multiple Tasks and Unlabeled Data

Rie Kubota Ando, Tong Zhang - 2005

10 papers in library cite

Very nice and clever way of solving the problem of semi-supervised learning, and makes a lot of sense. I give them more credit for formalizing the concept. The methodology is a bit boring.

[18]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[19]Grammar as a Foreign Language

Geoffrey Hinton - 2015

9 papers in library cite

It's a nice paper showing that attention can be used for parsing. However, parsing is boring and is very derivative. Good paper nonetheless.

[20]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[21]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[22]Unsupervised Learning of Video Representations Using LSTMS

N. Srivastava, E. Mansimov, Ruslan Salakhutdinov - 2015

3 papers in library cite

Video representation

[23]Beyond Short Snippets: Deep Networks for Video Classification

J. Y. H. Ng, M. J. Hausknecht, S. Vijayanarasimhan, Oriol Vinyals, R. Monga, G. Toderici - 2015

1 paper in library cites

NNs for video classificatio nsounds nice

[24]Neural Responding Machine for Short-Text Conversation

L. Shang, Z. L. Lu, H. Li - 2015

2 papers in library cite

Conversation bots

[25]Effective Use of Word Order for Text Categorization With Convolutional Neural Networks

K. R. Johnson, Tong Zhang - 2014

1 paper in library cites

Apparently the first to use cnn for text

[26]Listen, Attend and Spell

W. Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals - 2015

4 papers in library cite

Speech recog + attention

[27]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[28]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[29]Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Shijie Wang, Manning, C. Christopher - 2012

7 papers in library cite

[30]Semantic Compositionality Through Recursive Matrix-Vector Spaces

Richard Socher, B. Huval, Christopher D. Manning, Andrew Y. Ng - 2012

7 papers in library cite

[31]Convolutional Deep Belief Networks on cifar-10

Alex Krizhevsky - 2010

3 papers in library cite

[32]DBpedia – A Large-Scale, Multilingual Knowledge Base Extracted From Wikipedia

J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. N. Mendes, S. Hellmann, M. Morsey, P. V. Kleef, S. Auer - 2014

2 papers in library cite

[33]End-to-End Continuous Speech Recognition Using Attention-Based Recurrent Nn: First Results

J. Chorowski, D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

2 papers in library cite

[34]Hidden Factors and Hidden Topics: Understanding Rating Dimensions With Review Text

J. Mcauley, J. Leskovec - 2013

2 papers in library cite

[35]Datasets for Single-Label Text Categorization

A. C. Cachopo - 2015

1 paper in library cites

[36]Learning Algorithms for the Classification Restricted Boltzmann Machine

Hugo Larochelle, M. Mandel, Razvan Pascanu, Yoshua Bengio - 2012

1 paper in library cites

[37]Newsweeder: Learning to Filter Netnews

K. Lang - 1995

1 paper in library cites

[38]Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs

Yann Dauphin, Yoshua Bengio - 2013

1 paper in library cites

Cited by

27

papers in your library

Cites

27

papers in your library

Read

on October 17, 2025

Very good paper that was probably the first to introduce pre-training in NLP!

Tags

Paper Aliases

No aliases