Papperoni

2017

Enriching Word Vectors With Subword Information

Tomas Mikolov

Open PDF Google Scholar

citations

Cite Score

89

AI summary

This paper introduces an extension of the continuous skip-gram model that represents words as the sum of character n-grams, enabling the learning of word representations that take into account subword information. The model is evaluated on nine languages and achieves state-of-the-art results on word similarity and analogy tasks.

Main Contributions

Introduces a new approach to word representation based on character n-grams.
Extends the continuous skip-gram model to incorporate subword information.
Demonstrates that the model can effectively capture morphological similarities between words.
Achieves state-of-the-art results on word similarity and analogy tasks across nine languages.
Shows the model is capable of computing representations for out-of-vocabulary words.

Abstract

Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Popular models that learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skipgram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram; words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpora quickly and allows us to compute word representations for words that did not appear in the training data. We evaluate our word representations on nine different languages, both on word similarity and analogy tasks. By comparing to recently proposed morphological word representations, we show that our vectors achieve state-of-the-art performance on these tasks.

Citation Graph

Loading graph...

References [40]

Sort:

Filter:

[1]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[4]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[5]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[6]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[7]Indexing by Latent Semantic Analysis

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman - 1990

12 papers in library cite

LSA paper

[8]Character-Level Convolutional Networks for Text Classification

X. Zhang, J. Zhao, Yann Lecun - 2015

7 papers in library cite

LeCun + Character CNNs

[9]From Frequency to Meaning: Vector Space Models of Semantics

P. D. Turney, P. Pantel - 2010

6 papers in library cite

[10]Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Benjamin Recht, C. Re, S. Wright, F. Niu - 2011

6 papers in library cite

[11]Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

W. Ling, C. Dyer, A. W. Black, I. Trancoso, R. Fermandez, S. Amir, L. Marujo, T. Luis - 2015

5 papers in library cite

[12]Better Word Representations With Recursive Neural Networks for Morphology

T. Luong, Richard Socher, Christopher D. Manning - 2013

4 papers in library cite

[13]Achieving Open Vocabulary Neural Machine Translation With Hybrid Word-Character Models

M. T. Luong, Christopher D. Manning - 2016

3 papers in library cite

[14]Improved Transition-Based Parsing by Modeling Characters Instead of Words With LSTMs

M. Ballesteros, C. Dyer, Noah A. Smith - 2015

3 papers in library cite

[15]Producing High-Dimensional Semantic Spaces From Lexical Co-Occurrence

K. Lund, C. Burgess - 1996

3 papers in library cite

Hinrich Schutze - 1993

3 papers in library cite

[17]CHARAGRAM: Embedding Words and Sentences via Character N-Grams

J. Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu - 2016

2 papers in library cite

[18]Compositional Morphology for Word Representations and Language Modelling

J. A. Botha, Phil Blunsom - 2014

2 papers in library cite

[19]Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts

C. N. D. Santos, M. Gatti - 2014

2 papers in library cite

[20]Distributional Memory: A General Framework for Corpus-Based Semantics

M. Baroni, A. Lenci - 2010

2 papers in library cite

[21]Factored Neural Language Models

A. Alexandrescu, K. Kirchhoff - 2006

2 papers in library cite

[22]The Westbury Lab Wikipedia Corpus

C. Shaoul, C. Westbury - 2010

2 papers in library cite

[23]Alternative Structures for Character-Level RNNs

Piotr Bojanowski, Armand Joulin, Tomas Mikolov - 2015

1 paper in library cites

[24]Automatically Creating Datasets for Measures of Semantic Relatedness

T. Zesch, I. Gurevych - 2006

1 paper in library cites

[25]Co-Learning of Word Representations and Morpheme Representations

S. Qiu, Q. Cui, J. Bian, B. Gao, T. Y. Liu - 2014

1 paper in library cites

[26]Compositionally Derived Representations of Morphologically Complex Words in Distributional Semantics

A. Lazaridou, Marco Marelli, R. Zamparelli, M. Baroni - 2013

1 paper in library cites

[27]Dimensions of Meaning

Hinrich Schutze - 1992

1 paper in library cites

[28]Human and Machine Judgements for Russian Semantic Relatedness

A. Panchenko, D. Ustalov, N. Arefyev, D. Paperno, N. Konstantinova, N. Loukachevitch, C. Biemann - 2016

1 paper in library cites

[29]Joint Learning of Character and Word Embeddings

X. Chen, L. Xu, Ze Liu, Maosong Sun, H. Luan - 2015

1 paper in library cites

[30]KNET: A General Framework for Learning Word Embedding Using Morphological Knowledge

Q. Cui, B. Gao, J. Bian, S. Qiu, H. Dai, T. Y. Liu - 2015

1 paper in library cites

[31]Learning Character-Level Representations for Part-of-Speech Tagging

C. N. D. Santos, B. Zadrozny - 2014

1 paper in library cites

[32]Letter N-Gram-Based Input Encoding for Continuous Space Language Models

H. Sperr, J. Niehues, A. Waibel - 2013

1 paper in library cites

[33]Morphological Word-Embeddings

R. Cotterell, Hinrich Schutze - 2015

1 paper in library cites

[34]Morphology-Based and Sub-Word Language Modeling for Turkish Speech Recognition

H. Sak, M. Saraclar, T. Gungor - 2010

1 paper in library cites

[35]Multilingual Reliability and Semantic Structure of Continuous Word Spaces

M. Koper, C. Scheible, S. S. I. Walde - 2015

1 paper in library cites

[36]Neurocomputing: Foundations of Research. Chapter Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1988

1 paper in library cites

[37]New Word Analogy Corpus for Exploring Embeddings of Czech Words

L. Svoboda, T. Brychcin - 2016

1 paper in library cites

[38]Normalizing Tweets With Edit Scripts and Recurrent Neural Embeddings

G. Chrupala - 2014

1 paper in library cites

[39]The Proof and Measurement of Association Between Two Things

C. Spearman - 1904

1 paper in library cites

[40]Unsupervised Morphology Induction Using Word Embeddings

Radu Soricut, F. Och - 2015

1 paper in library cites

Cited by

7

papers in your library

Cites

8

papers in your library

Read

on November 27, 2025

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

Tags

Paper Aliases

No aliases