Papperoni

2013

Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever

Open PDF Google Scholar

citations

Cite Score

54

AI summary

This paper introduces a method for machine translation that automates the process of generating dictionaries and phrase tables by learning a linear projection between vector spaces of languages using distributed word representations, achieving almost 90% precision@5 for translation of words between English and Spanish.

Main Contributions

Introduces a method that can automate the process of generating and extending dictionaries and phrase tables.
Translates missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data.
Uses distributed representation of words and learns a linear mapping between vector spaces of languages.
Achieves almost 90% precision@5 for translation of words between English and Spanish.
The method makes little assumption about the languages, so it can be used to extend and refine dictionaries and translation tables for any language pairs.

Abstract

Dictionaries and phrase tables are the basis of modern statistical machine translation systems. This paper develops a method that can automate the process of generating and extending dictionaries and phrase tables. Our method can translate missing word and phrase entries by learning language structures based on large monolingual data and mapping between languages from small bilingual data. It uses distributed representation of words and learns a linear mapping between vector spaces of languages. Despite its simplicity, our method is surprisingly effective: we can achieve almost 90% precision@5 for translation of words between English and Spanish. This method makes little assumption about the languages, so it can be used to extend and refine dictionaries and translation tables for any language pairs.

Citation Graph

Loading graph...

References [19]

Sort:

Filter:

[1]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[4]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[5]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[6]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[7]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[8]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[9]Linguistic Regularities in Continuous Space Word Representations

Tomas Mikolov, W. T. Yih, Geoffrey Zweig - 2013

8 papers in library cite

This builds on Bengio's "Word Representations" paper to show that embeddings can be used to describe syntactic and semantic characteristics of words. Good contribution, but not sure if too impactful.

[10]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[11]Parsing Natural Scenes and Natural Language With Recursive Neural Networks

Richard Socher, C. C. Lin, C. Manning, Andrew Y. Ng - 2011

10 papers in library cite

Good idea and nice results, but not my thing right now.

[12]Improving Word Representations via Global Context and Multiple Word Prototypes

Eric H. Huang, Richard Socher, C. Manning, Andrew Y. Ng - 2012

7 papers in library cite

Good paper. I would say not too relevant, but has some nice concepts like a "global context vector" (document embedding) that comes from averaging word embeddings, and a joint training objective.

[13]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

Nice paper overall. Seems very impactful despite not being as relevant right now.

[14]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[15]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[16]Parsing With Compositional Vector Grammars

Richard Socher, J. Bauer, Christopher D. Manning, Andrew Y. Ng - 2013

3 papers in library cite

[17]Learning a Translation Lexicon From monolingual corpora

P. Koehn, K. Knight - 2002

2 papers in library cite

[18]Learning Bilingual Lexicons From monolingual corpora

A. Haghighi, Percy Liang, T. B. Kirkpatrick, Dan Klein - 2008

2 papers in library cite

[19]Estimating Word Translation Probabilities From Unrelated Monolingual corpora Using the em algorithm

P. Koehn, K. Knight - 2000

1 paper in library cites

Cited by

6

papers in your library

Cites

14

papers in your library

Read

on November 27, 2025

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

Tags

Paper Aliases

No aliases