Papperoni

2014

Convolutional Neural Networks for Sentence Classification

Yoon Kim

Open PDF Google Scholar

citations

Cite Score

91

AI summary

This paper introduces a CNN model for sentence classification using pre-trained word vectors, achieving state-of-the-art results on multiple benchmarks including sentiment analysis and question classification. The model uses static and task-specific vectors and a simple modification to use both types of vectors.

Main Contributions

Demonstrates that a simple CNN with static word vectors achieves excellent results on multiple benchmarks for sentence classification.
Shows that learning task-specific vectors through fine-tuning offers further gains in performance.
Proposes a simple modification to the architecture to allow for the use of both task-specific and static vectors.
Achieves state-of-the-art results on 4 out of 7 tasks, including sentiment analysis and question classification, surpassing existing deep learning models.

Abstract

We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.

Citation Graph

Loading graph...

References [31]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[5]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[6]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[7]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[8]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[9]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[10]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[11]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[12]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[13]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[14]CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

A. Razavian, H. Azizpour, J. Sullivan, S. Carlsson - 2014

6 papers in library cite

CNN feature extraction

[15]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[16]A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Bo Pang, L. A. Lee, L. Lillian - 2004

8 papers in library cite

[17]Annotating Expressions of Opinions and Emotions in Language

J. Wiebe, T. Wilson, T. Theresa, C. A. Cardie, C. Claire - 2005

7 papers in library cite

[18]Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Shijie Wang, Manning, C. Christopher - 2012

7 papers in library cite

[19]Semantic Compositionality Through Recursive Matrix-Vector Spaces

Richard Socher, B. Huval, Christopher D. Manning, Andrew Y. Ng - 2012

7 papers in library cite

[20]Mining and Summarizing Customer Reviews

M. Hu, B. A. Liu, B. Bing - 2004

6 papers in library cite

[21]Fast Dropout Training

Shijie Wang, C. Manning - 2013

4 papers in library cite

[22]Dependency Tree-Based Sentiment Classification Using CRFs With Hidden Variables

T. Nakagawa, K. Inui, S. Kurohashi - 2010

3 papers in library cite

[23]From Symbolic to Sub-Symbolic Information in Question Classification

J. P. C. G. D. Silva, L. Coheur, A. C. Mendes, A. Wichert - 2011

3 papers in library cite

[24]Learning Question Classifiers

Xiang Lisa Li, Dan Roth - 2002

3 papers in library cite

[25]The Role of Syntax in Vector Space Models of Compositional Semantics

K. Hermann, Phil Blunsom - 2013

3 papers in library cite

[26]Semantic Parsing for Single-Relation Question Answering

W. Yih, X. He, C. Meek - 2014

2 papers in library cite

[27]A Statistical Parsing Framework for Sentiment Classification

L. Dong, F. Wei, Shuming Liu, M. Zhou, K. Xu - 2014

1 paper in library cites

[28]Context-Aware Learning for Sentence-Level Sentiment Analysis With Posterior Regularization

B. Yang, C. Cardie - 2014

1 paper in library cites

[29]Learning Discriminative Projections for Text Similarity Measures

W. Yih, Kristina Toutanova, J. Platt, C. Meek - 2011

1 paper in library cites

[30]Learning Semantic Representations Using Convolutional Neural Networks for Web Search

Y. Shen, X. He, Jianfeng Gao, L. Deng, G. Mesnil - 2014

1 paper in library cites

[31]Political Ideology Detection Using Recursive Neural Networks

M. Iyyer, P. Enns, J. B. Graber, P. Resnik - 2014

1 paper in library cites

Cited by

8

papers in your library

Cites

14

papers in your library

Read

on October 20, 2025

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

Tags

Paper Aliases

No aliases