Papperoni

2019

No Training Required: Exploring Random Encoders for Sentence Classification

J. Wieting, Douwe Kiela

Open PDF Google Scholar

citations

Cite Score

7

AI summary

This paper explores random parameterizations for sentence representations from pre-trained word embeddings. It shows that modern sentence embeddings gain surprisingly little over random methods. The paper provides more appropriate baselines for sentence classification evaluation and makes observations about proper experimental protocol.

Main Contributions

Explores various methods for computing sentence representations from pre-trained word embeddings without any training.
Demonstrates that modern sentence embeddings gain surprisingly little over random methods.
Provides the field with more appropriate baselines for sentence classification evaluation.
Makes important observations about proper experimental protocol for sentence classification evaluation.
Offers recommendations for future research.

Abstract

We explore various methods for computing sentence representations from pretrained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward—which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.

Citation Graph

Loading graph...

References [61]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[3]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[4]Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun - 2015

10 papers in library cite

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

[5]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[6]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[7]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[8]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[9]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[10]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[11]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[12]What Is the Best Multi-Stage Architecture for Object Recognition?

K. Jarrett, Koray Kavukcuoglu, Marc'aurelio Ranzato, Yann Lecun - 2009

20 papers in library cite

So boring and convoluted... They try to make things more abstract by saying "filter banks" and "2 stages", but in the end it's CNNs. It seems like they are trying to tie NNs to the rest of ML, but it doesn't work.

[13]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[14]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[15]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[16]Troubling Trends in Machine Learning Scholarship

Jacob Steinhardt - 2018

1 paper in library cites

Amazing push and describes much of what I've read!

[17]Deep Image Prior

D. Ulyanov, A. Vedaldi, Victor Lempitsky - 2017

1 paper in library cites

Random image baseline

[18]A Simple but Tough-to-Beat Baseline for Sentence Embeddings

S. Arora, Yiqing Liang, T. Ma - 2017

4 papers in library cite

[19]Universal Sentence Encoder

D. Cer, Yining Yang, S. Y. Kong, N. Hua, N. Limtiaco, R. S. John, Noah Constant, M. G. Cespedes, S. Yuan, C. Tar, Y. H. Sung, B. Strope, R. Kurzweil - 2018

2 papers in library cite

[20]What You Can Cram Into a Single Vector: Probing Sentence Embeddings for Linguistic Properties

Alexis Conneau, German Kruszewski, G. Lample, L. Barrault, M. Baroni - 2018

2 papers in library cite

[21]Senteval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela - 2018

5 papers in library cite

Sentence embedding eval toolkit?

[22]Towards Universal Paraphrastic Sentence Embeddings

J. Wieting, Mohit Bansal, Kevin Gimpel, K. A. Livescu, K. Karen - 2015

4 papers in library cite

[23]What Do Neural Machine Translation Models Learn About Morphology?

Yonatan Belinkov, N. Durrani, F. Dalvi, H. Sajjad, J. R. Glass - 2017

2 papers in library cite

They perform an "intrinsic evaluation" of words to understand what they represent

[24]Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

D. Shen, Gloria Wang, Wenyi Wang, M. R. Min, Q. Su, Y. Z. Zhang, Chun-Liang Li, R. Henao, L. Carin - 2018

1 paper in library cites

[25]Learning General Purpose Distributed Sentence Representations via Large Scale Multi-Task Learning

S. Subramanian, A. Trischler, Yoshua Bengio, C. Pal - 2018

4 papers in library cite

Bengio

[26]Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Shijie Wang, Manning, C. Christopher - 2012

7 papers in library cite

[27]Assessing the Ability of lstms to Learn Syntax-Sensitive Dependencies

Tal Linzen, E. Dupoux, Y. Goldberg - 2016

5 papers in library cite

[28]Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Yacine Jernite, S. Bowman, D. Sontag - 2017

4 papers in library cite

[29]DisSent: Learning Sentence Representations From Explicit Discourse Relations

Allen Nie, E. Bennett, N. Goodman - 2017

4 papers in library cite

[30]Fine-Grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

Y. Adi, E. Kermany, Yonatan Belinkov, O. Lavi, Y. Goldberg - 2016

4 papers in library cite

[31]Perceptron: An Introduction to Computational Geometry

M. Minsky, S. Papert - 1969

4 papers in library cite

[32]How Grammatical Is Character-Level Neural Machine Translation? Assessing Mt quality With Contrastive Translation Pairs

R. Sennrich - 2016

2 papers in library cite

[33]Jointly Optimizing Word Representations for Lexical and Sentential Tasks With the C-Phrase Model

N. T. Pham, German Kruszewski, A. Lazaridou, M. Baroni - 2015

2 papers in library cite

[34]Learning Visually Grounded Sentence Representations

Douwe Kiela, Alexis Conneau, A. Jabri, M. Nickel - 2017

2 papers in library cite

[35]Reservoir Computing Approaches to Recurrent Neural Network Training

M. Lukosevicius, Herbert Jaeger - 2009

2 papers in library cite

[36]Weighted Sums of Random Kitchen Sinks: Replacing Minimization With Randomization in Learning

A. Rahimi, Benjamin Recht - 2009

2 papers in library cite

[37]An Outline of a Mathematical Theory of Papa

A. Borsellino, A. Gamba - 1961

1 paper in library cites

[38]Exploring Semantic Properties of Sentence Embeddings

X. Zhu, Tao Li, G. Melo - 2018

1 paper in library cites

[39]Extreme Learning Machine: Theory and Applications

G. B. Huang, Q. Y. Zhu, C. K. Siew - 2006

1 paper in library cites

[40]Feedforward Neural Networks With Random Weights

W. F. Schmidt, M. A. Kraaijveld, R. P. Duin - 1992

1 paper in library cites

[41]Further Experiments With Papa

A. Gamba, L. Gamberini, G. Palmieri, R. Sanna - 1961

1 paper in library cites

[42]Generalization Without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks

B. M. Lake, M. Baroni - 2018

1 paper in library cites

[43]Geometrical and Statistical Properties of Systems of Linear Inequalities With Applications in Pattern Recognition

T. M. Cover - 1965

1 paper in library cites

[44]Learn More by Training Less: Systematicity in Sentence Processing by Recurrent Networks

S. L. Frank - 2006

1 paper in library cites

[45]Learning and Generalization Characteristics of the Random Vector Functional-Link Net

Y. H. Pao, G. H. Park, D. J. Sobajic - 1994

1 paper in library cites

[46]Learning Grammatical Structure With echo State Networks

M. H. Tong, A. D. Bickett, E. M. Christiansen, G. W. Cottrell - 2007

1 paper in library cites

[47]Model-Free Pomdp Optimisation of Tutoring Systems With Echo-State Networks

L. Daubigney, M. Geist, O. Pietquin - 2013

1 paper in library cites

[48]On Random Weights and Unsupervised Feature Learning

Andrew M. Saxe, P. W. Koh, Ziru Chen, M. Bhand, B. Suresh, Andrew Y. Ng - 2011

1 paper in library cites

[49]On the Capabilities of Multilayer Perceptrons

E. B. Baum - 1988

1 paper in library cites

[50]On-Line Processing of Grammatical Structure Using Reservoir Computing

X. Hinaut, P. F. Dominey - 2012

1 paper in library cites

[51]Probing for Semantic Evidence of Composition by Means of Simple Classification Tasks

A. Ettinger, A. Elgohary, P. Resnik - 2016

1 paper in library cites

[52]Pushing the Limits of Paraphrastic Sentence Embeddings With Millions of Machine Translations

J. Wieting, Kevin Gimpel - 2017

1 paper in library cites

[53]Random Features for Large-Scale Kernel Machines

A. Rahimi, Benjamin Recht - 2008

1 paper in library cites

[54]Random Matrices

M. L. Mehta - 2004

1 paper in library cites

[55]Scalable Kernel Methods via Doubly Stochastic Gradients

B. Dai, B. Xie, N. He, Yiqing Liang, A. Raj, M. F. F. Balcan, L. Song - 2014

1 paper in library cites

[56]Strong Systematicity in Sentence Processing by an echo State Network

S. L. Frank - 2006

1 paper in library cites

[57]The “echo State” Approach to Analysing and Training Recurrent Neural Networks-With an Erratum Note

Herbert Jaeger - 2001

1 paper in library cites

[58]The Random Projection Method

S. S. Vempala - 2005

1 paper in library cites

[59]Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder

F. Dalvi, N. Durrani, H. Sajjad, Yonatan Belinkov, S. Vogel - 2017

1 paper in library cites

[60]Unsupervised Learning of Sentence Embeddings Using Compositional n-gram Features

M. Pagliardini, P. Gupta, M. Jaggi - 2017

1 paper in library cites

[61]Visualisation and’diagnostic Classifiers’ Reveal How Recurrent and Recursive Neural Networks Process Hierarchical Structure

Dieuwke Hupkes, S. Veldhoen, W. Zuidema - 2018

1 paper in library cites

Cited by

1

papers in your library

Cites

25

papers in your library

Read

on November 13, 2025

It's very amazing what they can do with random projections. Plus, it's a good push for better baselines.

Tags

Paper Aliases

No aliases