Papperoni

2014

A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner

Open PDF Google Scholar

citations

Cite Score

74

AI summary

This paper introduces the Dynamic Convolutional Neural Network (DCNN) for semantic sentence modeling, utilizing Dynamic k-Max Pooling to handle varying sentence lengths and capture short and long-range relations. The DCNN is tested on sentiment prediction, question classification, and Twitter sentiment prediction, achieving excellent performance.

Main Contributions

Introduces the Dynamic Convolutional Neural Network (DCNN) architecture for sentence modeling.
Presents Dynamic k-Max Pooling, a global pooling operation for linear sequences that handles variable-length inputs.
The DCNN induces a feature graph over the sentence that explicitly captures short and long-range relations.
The model is applicable to any language without relying on parse trees.
Demonstrates excellent performance on sentiment prediction, question classification, and Twitter sentiment prediction tasks.

Abstract

The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline.

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[2]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[3]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[4]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[5]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[6]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[7]Connectionist Learning Procedures

Geoffrey E. Hinton - 1987

11 papers in library cite

It's a very good overview of everything that was happening in 1987! A bit too long though, but a good start nonetheless.

[8]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[9]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[10]Extensions of Recurrent Neural Network Language Model

Tomas Mikolov, S. Kombrink, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2011

16 papers in library cite

Doesn't add much.

[11]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

Nothing too interesting, just using the context of the RNN.

[12]Continuous Space Translation Models for Phrase-Based Statistical Machine Translation

Holger Schwenk - 2012

5 papers in library cite

Nothing special.

[13]Recursive Distributed Representations

J. B. Pollack - 1990

7 papers in library cite

[14]Composition in Distributional Models of Semantics

J. Mitchell, Mirella Lapata - 2010

5 papers in library cite

[15]Grounded Compositional Semantics for Finding and Describing Images With Sentences

Richard Socher, Quoc Le, C. Manning, A. Ng - 2014

5 papers in library cite

[16]A Structured Vector Space Model for Word Meaning in Context

K. Erk, S. Pado - 2008

3 papers in library cite

[17]Estimating Linear Models for Compositional Distributional Semantics

F. Zanzotto, I. Korkontzelos, F. Fallucchi, S. Manandhar - 2010

3 papers in library cite

[18]Experimental Support for a Categorical Compositional Distributional Model of Meaning

Edward Grefenstette, M. Sadrzadeh - 2011

3 papers in library cite

[19]From Symbolic to Sub-Symbolic Information in Question Classification

J. P. C. G. D. Silva, L. Coheur, A. C. Mendes, A. Wichert - 2011

3 papers in library cite

[20]Learning Question Classifiers

Xiang Lisa Li, Dan Roth - 2002

3 papers in library cite

[21]LSTM recurrent Networks Learn Simple Context-Free and Context-Sensitive Languages

Felix A. Gers, Jürgen Schmidhuber - 2001

3 papers in library cite

[22]The Role of Syntax in Vector Space Models of Compositional Semantics

K. Hermann, Phil Blunsom - 2013

3 papers in library cite

[23]Vector-Based Models of Semantic Composition

J. Mitchell, Mirella Lapata - 2008

3 papers in library cite

[24]Domain and Function: A Dual-Space Model of Semantic Relations and Compositions

P. D. Turney - 2012

2 papers in library cite

[25]Learning to Map Sentences to Logical Form: Structured Classification With Probabilistic Categorial Grammars

L. S. Zettlemoyer, Michael Collins - 2005

2 papers in library cite

[26]Nouns Are Vectors, Adjectives Are Matrices: Representing Adjective-Noun Constructions in Semantic Space

M. Baroni, R. Zamparelli - 2010

2 papers in library cite

[27]Recurrent Convolutional Neural Networks for Discourse Compositionality

N. Kalchbrenner, Phil Blunsom - 2013

2 papers in library cite

[28]A Context-Theoretic Framework for Compositionality in Distributional Semantics

D. Clarke - 2012

1 paper in library cites

[29]Category-Theoretic Quantitative Compositional Distributional Models of Natural Language Semantics

Edward Grefenstette - 2013

1 paper in library cites

[30]Inductive Learning in Symbolic Domains Using Structure-Driven Recurrent Neural Networks

A. Kuchler, C. Goller - 1996

1 paper in library cites

[31]Mathematical Foundations for a Compositional Distributional Model of Meaning

B. Coecke, M. Sadrzadeh, S. Clark - 2010

1 paper in library cites

[32]Modelling Adjective-Noun Compositionality by Regression

E. Guevara - 2010

1 paper in library cites

[33]Prior Disambiguation of Word Tensors for Constructing Sentence Vectors

D. Kartsaklis, M. Sadrzadeh - 2013

1 paper in library cites

[34]Question Classification Using Head Words and Their Hypernyms

Zhongqiang Huang, M. Thint, Z. Qin - 2008

1 paper in library cites

[35]Question Classification With Log-Linear Models

Phil Blunsom, K. Kocik, J. R. Curran - 2006

1 paper in library cites

[36]Readings in Speech Recognition. Chapter Phoneme Recognition Using Time-Delay Neural Networks

A. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, Kevin J. Lang - 1990

1 paper in library cites

[37]Twitter Sentiment Classification Using Distant Supervision

A. Go, R. Bhayani, L. Huang - 2009

1 paper in library cites

[38]Vector Space Models of Word Meaning and Phrase Meaning: A Survey

K. Erk - 2012

1 paper in library cites

Cited by

7

papers in your library

Cites

13

papers in your library

Read

on December 23, 2025

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

Tags

Paper Aliases

No aliases