Papperoni

2017

Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever

Open PDF Google Scholar

citations

Cite Score

25

AI summary

This paper introduces a byte-level recurrent language model that learns disentangled features, including a sentiment analysis unit, by training on a large Amazon product review dataset. It achieves state-of-the-art results on the Stanford Sentiment Treebank and demonstrates data efficiency with few labeled examples.

Main Contributions

Discovered a single unit within a byte-level mLSTM that directly corresponds to sentiment.
Achieved state-of-the-art results on the binary subset of the Stanford Sentiment Treebank in an unsupervised manner.
Demonstrated data efficiency by matching performance of strong baselines trained on full datasets using only a handful of labeled examples.
Showed the sentiment unit has a direct influence on the generative process of the model.
Visualized the value of the sentiment cell as it processes reviews.

Abstract

We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.

Citation Graph

Loading graph...

References [56]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]Deep Learning

I. Goodfellow, Yoshua Bengio, Y. A. Courville, A. Aaron - 2016

5 papers in library cite

It's an awesome review for people that are not familiar with deep learning, but it's very basic as well. Not too relevant given the context of other papers I read, but huge impact in terms of scientific communication.

[4]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[5]Reducing the Dimensionality of Data With Neural Networks

Geoffrey Hinton, Ruslan Salakhutdinov - 2006

37 papers in library cite

I didn't like the way this is written, very hard to understand without a ton of background knowledge. But hey, it's the first deep learning model!

[6]Visualizing and Understanding Convolutional Networks

Matthew D. Zeiler, Rob Fergus - 2014

15 papers in library cite

Very good explanation and visualization of CNNs, and also nice that they use their findings to improve the performance. The ablation study is also nice.

[7]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[8]Representation Learning: A Review and New Perspectives

Yoshua Bengio, Aaron Courville, A. A. Vincent, P. Pascal - 2013

4 papers in library cite

Good in spirit but is too long and covers a lot of things that are not relevant anymore.

[9]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[10]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[11]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[12]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[13]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[14]Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation

Yonghui Wu, M. Schuster, Ziru Chen, Quoc V. Le, M. Norouzi, W. Macherey, M. Krikun, Yue Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. J. Johnson, Xiaodong Liu, Lukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, Wenyi Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, Oriol Vinyals, G. S. Corrado, M. Hughes, Jeffrey Dean - 2016

15 papers in library cite

It's a very good paper but TBH doesn't bring anything new other than joining a bunch of existing stuff. I think it ended up being foundational because it's Google and several people used it as a base for future research. Good contribution then :)

[15]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[16]Linguistic Regularities in Continuous Space Word Representations

Tomas Mikolov, W. T. Yih, Geoffrey Zweig - 2013

8 papers in library cite

This builds on Bengio's "Word Representations" paper to show that embeddings can be used to describe syntactic and semantic characteristics of words. Good contribution, but not sure if too impactful.

[17]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[18]Building High-Level Features Using Large Scale Unsupervised Learning

Quoc V. Le, M. A. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, Jeffrey Dean, Andrew Y. Ng - 2012

10 papers in library cite

Very nice and very early work - seems very simple but very insightful to use an autoencoder to detect objects. Also, very similar to the neocognitron :)

[19]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[20]Unsupervised Learning of Invariant Feature Hierarchies With Applications to Object Recognition

Marc'aurelio Ranzato, F. Huang, Y. Boureau, Yann Lecun - 2007

8 papers in library cite

I thought that I had learned pretty much everything before 2015, but this paper proved me wrong. Amazing read and very nice methodology. Very simple as well!

[21]Visualizing and Understanding Recurrent Networks

Li Fei Fei - 2015

3 papers in library cite

Very nice! The results were a bit underwhelming though. Nice visualizations but few explanations. Good nonetheless.

[22]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[23]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[24]Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups

Geoffrey E. Hinton, L. Deng, D. Yu, George E. Dahl, A. Mohamed, Navdeep Jaitly, A. Senior, Vincent Vanhoucke, P. Nguyen, T. N. Sainath, Brian Kingsbury - 2012

8 papers in library cite

The core of the paper itself is a bit boring and doesn't introduce anything new (just RBMs and DBNs again) but I am giving this a 4 because it's probably the best explanation of RBMs and DBNs I've read so far.

Reference title contains 'et al'

[25]Latent Dirichlet Allocation

D. M. Blei, Andrew Y. Ng, Michael I. Jordan - 2003

10 papers in library cite

30 pages; LDA

[26]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[27]Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

K. S. Tai, Richard Socher, Christopher D. Manning - 2015

6 papers in library cite

Baseline for skip-thoughts paper

[28]Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

T. Salimans, D. A. Kingma, D. P. Diederik - 2016

4 papers in library cite

Cited by a lot of people

[29]Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, Victor Zhong, R. Paulus, Richard Socher - 2015

9 papers in library cite

[30]Object Detectors Emerge in Deep Scene CNNS

B. Zhou, A. Khosla, A. Aditya, A. Lapedriza, A. Agata, Aude Oliva, A. A. Torralba, A. Antonio - 2014

1 paper in library cites

Interesting

[31]Towards Universal Paraphrastic Sentence Embeddings

J. Wieting, Mohit Bansal, Kevin Gimpel, K. A. Livescu, K. Karen - 2015

4 papers in library cite

[32]Unsupervised Construction of Large Paraphrase corpora: Exploiting Massively Parallel News Sources

B. Dolan, C. Quirk, C. A. Brockett, C. Chris - 2004

5 papers in library cite

Seems very very close to the other paper I read, which I found boring. Probably not worth the read.

[33]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[34]Sparse Coding With an Overcomplete Basis Set: A Strategy Employed by V1?

Bruno A. Olshausen, David J. Field - 1997

10 papers in library cite

[35]A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Bo Pang, L. A. Lee, L. Lillian - 2004

8 papers in library cite

[36]An Analysis of Single-Layer Networks in Unsupervised Feature Learning

A. Coates, A. Ng, Honglak Lee - 2011

7 papers in library cite

[37]Annotating Expressions of Opinions and Emotions in Language

J. Wiebe, T. Wilson, T. Theresa, C. A. Cardie, C. Claire - 2005

7 papers in library cite

[38]Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Shijie Wang, Manning, C. Christopher - 2012

7 papers in library cite

[39]Semeval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences Through Semantic Relatedness and Textual Entailment

Marco Marelli, L. Bentivogli, M. Baroni, R. Bernardi, S. Menini, R. Zamparelli - 2014

7 papers in library cite

[40]Mining and Summarizing Customer Reviews

M. Hu, B. A. Liu, B. Bing - 2004

6 papers in library cite

[41]Adversarial Training Methods for Semi-Supervised Text Classification

T. Miyato, A. M. Dai, I. Goodfellow - 2016

4 papers in library cite

[42]Learning and Transferring Mid-Level Image Representations Using Convolutional Neural Networks

Maxime Oquab, Leon Bottou, I. Laptev, Josef Sivic - 2014

4 papers in library cite

[43]The Human Knowledge Compression Contest

M. Hutter - 2012

4 papers in library cite

[44]A Fast and Accurate Dependency Parser Using Neural Networks

Deli Chen, C. Manning - 2014

3 papers in library cite

[45]Multiplicative LSTM for Sequence Modelling

B. Krause, L. Lu, I. Murray, S. Renals - 2016

3 papers in library cite

[46]Self-Adaptive Hierarchical Sentence Model

H. Zhao, Z. L. Lu, Z. A. Poupart, P. Pascal - 2015

3 papers in library cite

[47]Topicrnn: A Recurrent Neural Network With Long-Range Semantic Dependency

A. Dieng, A. B., Caitlin Wang, Jianfeng Gao, J. A. Paisley, J. John - 2016

3 papers in library cite

[48]Deep Learning With Dynamic Computation Graphs

M. Looks, M. Herreshoff, M. Hutchins, D. A. Norvig, P. Peter - 2017

2 papers in library cite

[49]Feature Selection, 11 vs. 12 Regularization, and Rotational Invariance

A. Ng, A. Y. - 2004

2 papers in library cite

[50]Re-Examining Machine Translation Metrics for Paraphrase Identification

N. Madnani, J. Tetreault, J. A. Chodorow, M. M. K. Martin - 2012

2 papers in library cite

[51]Controllable Text Generation

Z. Hu, Zhilin Yang, Z. Zichao, X. Liang, Ruslan Salakhutdinov, R. A. Xing, E. P. Eric - 2017

1 paper in library cites

[52]Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews

G. Mesnil, Tomas Mikolov, T. Tomas, Ranzato, M. Marc'aurelio, Bengio, Y. Yoshua - 2014

1 paper in library cites

[53]Inferring Networks of Substitutable and Complementary Products

J. Mcauley, R. Pandey, R. Rahul, L. A. Leskovec, J. Jure - 2015

1 paper in library cites

[54]Learning to Generate Product Reviews From Attributes

L. Dong, S. Huang, S. Wei, F. Furu, Mirella Lapata, M. Mirella, M. Zhou, K. A. Xu, K. Ke - 2017

1 paper in library cites

[55]Understanding Neural Networks Through Representation Erasure

Jeffrey Li, W. Monroe, W. A. Jurafsky, D. Dan - 2016

1 paper in library cites

[56]Utf-8, a Transformation Format of iso 10646

F. Yergeau - 2003

1 paper in library cites

Cited by

8

papers in your library

Cites

31

papers in your library

Read

on November 5, 2025

Very impressive results. We can see that they were onto something (later became GPT2)

Tags

Paper Aliases

No aliases