Papperoni

2018

Word Translation Without Parallel Data

Alexis Conneau, G. Lample, Marc'aurelio Ranzato, L. Denoyer, Hervé Jégou

Open PDF Google Scholar

citations

Cite Score

45

AI summary

This paper introduces a new unsupervised approach that leverages adversarial training and cross-domain similarity local scaling to learn cross-lingual word embeddings, achieving state-of-the-art results on word translation, sentence translation retrieval, and cross-lingual word similarity tasks on several language pairs using fastText embeddings.

Main Contributions

Introduces an unsupervised approach for cross-lingual word embeddings that matches or exceeds supervised methods.
Introduces a cross-domain similarity adaptation to address the hubness problem.
Proposes an unsupervised criterion for model selection correlated with mapping quality.
Releases high-quality dictionaries for 12 oriented language pairs and corresponding word embeddings.
Demonstrates effectiveness on a low-resource language pair (English-Esperanto).

Abstract

State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.

Citation Graph

Loading graph...

References [52]

Sort:

Filter:

[1]Generative Adversarial Nets

Ian J. Goodfellow, J. P. Abadie, M. Mirza, B. Xu, D. W. Farley, S. Ozair, Aaron Courville, Yoshua Bengio - 2014

2 papers in library cite

The idea is surprisingly simple. I was a bit afraid of the math, but it's very simple. There's proof that this converges and finds an optimum. I just wish they explained more, or gave better results. It just seems that it finishes and I'm still hungry.

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[5]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[6]Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever - 2013

6 papers in library cite

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

[7]Domain-Adversarial Training of Neural Networks

Yaroslav Ganin, E. Ustinova, H. Ajakan, P. Germain, Hugo Larochelle, F. Laviolette, M. Marchand, Victor Lempitsky - 2016

1 paper in library cites

Adversarial domain adaptation?

[8]Billion-Scale Similarity Search With Gpus

J. Johnson, M. Douze, Hervé Jégou - 2017

4 papers in library cite

Efficient KNN with GPUs

[9]Nips 2016 Tutorial: Generative Adversarial Networks

I. Goodfellow - 2016

1 paper in library cites

50+ pages, but it's a tutorial!!!

[10]Neural Word Embeddings as Implicit Matrix Factorization

Omer Levy, Y. Goldberg - 2014

4 papers in library cite

Improve understanding of embeddings

[11]Bilingual Word Embeddings for Phrase-Based Machine Translation

W. Zou, Richard Socher, D. Cer, C. Manning - 2013

4 papers in library cite

Bilingual embeddings?

[12]Offline Bilingual Word Vectors, Orthogonal Transformations and the Inverted Softmax

S. L. Smith, D. H. Turban, S. Hamblin, N. Y. Hammerla - 2017

4 papers in library cite

Inverted softmax

[13]Learning Bilingual Word Embeddings With (Almost) No Bilingual Data

M. Artetxe, G. Labaka, E. Agirre - 2017

2 papers in library cite

Bilingual word embeddings

[14]Learning Principled Bilingual Mappings of Word Embeddings While Preserving Monolingual Invariance

M. Artetxe, G. Labaka, E. Agirre - 2016

2 papers in library cite

Bilingual word embeddings

[15]Adversarial Training for Unsupervised Bilingual lexicon Induction

Mingchuan Zhang, Yibo Liu, H. Luan, Maosong Sun - 2017

2 papers in library cite

Flopped a bit but adversarial leraning

[16]Hubness and Pollution: Delving Into Cross-Space Mapping for Zero-Shot Learning

A. Lazaridou, G. Dinu, M. Baroni - 2015

2 papers in library cite

The first to notice the "hubness" in multi dimensional spaces

[17]English Gigaword

R. Parker, D. Graff, J. Kong, K. Chen, K. Maeda - 2011

5 papers in library cite

[18]Better Word Representations With Recursive Neural Networks for Morphology

T. Luong, Richard Socher, Christopher D. Manning - 2013

4 papers in library cite

[19]Distributional Structure

Z. Harris - 1954

3 papers in library cite

[20]Bilbowa: Fast Bilingual Distributed Representations Without Word Alignments

S. Gouws, Yoshua Bengio, G. Corrado - 2015

2 papers in library cite

[21]Deciphering Foreign Language

S. Ravi, K. Knight - 2011

2 papers in library cite

[22]Improving Vector Space Word Representations Using Multilingual Correlation

Manaal Faruqui, C. Dyer - 2014

2 papers in library cite

[23]Learning a Translation Lexicon From monolingual corpora

P. Koehn, K. Knight - 2002

2 papers in library cite

[24]Learning Bilingual Lexicons From monolingual corpora

A. Haghighi, Percy Liang, T. B. Kirkpatrick, Dan Klein - 2008

2 papers in library cite

[25]Massively Multilingual Word Embeddings

W. Ammar, G. Mulcaire, Y. Tsvetkov, G. Lample, C. Dyer, Noah A. Smith - 2016

2 papers in library cite

[26]Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

C. Xing, D. Wang, C. L. Liu, Yutong Lin - 2015

2 papers in library cite

[27]Parallel Data, Tools and Interfaces in Opus

J. Tiedemann - 2012

2 papers in library cite

[28]Semeval-2017 Task 2: Multilingual and Cross-Lingual Semantic Word Similarity

J. C. Collados, M. T. Pilehvar, N. Collier, R. Navigli - 2017

2 papers in library cite

[29]The Earth Mover's Distance as a Metric for Image Retrieval

Y. Rubner, C. Tomasi, L. J. Guibas - 2000

2 papers in library cite

[30]Unifying Bayesian Inference and Vector Space Models for Improved Decipherment

Q. Dou, Ashish Vaswani, K. Knight, C. Dyer - 2015

2 papers in library cite

[31]A Distribution-Based Model to Learn Bilingual Word Embeddings

H. Cao, T. Z. Zhao, S. Zhang, Y. Meng - 2016

1 paper in library cites

[32]A Generalized Solution of the Orthogonal Procrustes Problem

P. H. Schonemann - 1966

1 paper in library cites

[33]Accurate Image Search Using the Contextual Dissimilarity Measure

Hervé Jégou, Cordelia Schmid, H. Harzallah, J. Verbeek - 2010

1 paper in library cites

[34]An Eigendecomposition Approach to Weighted Graph Matching Problems

S. Umeyama - 1988

1 paper in library cites

[35]An Ir Approach for Translating New Words From nonparallel, Comparable Texts

Pascale Fung, L. Y. Yee - 1998

1 paper in library cites

[36]Automatic Identification of Word Translations From Unrelated English and german corpora

R. Rapp - 1999

1 paper in library cites

[37]Bilingual Word Embeddings From Non-Parallel Document-Aligned Data Applied to Bilingual Lexicon Induction

I. Vulic, M. F. Moens - 2015

1 paper in library cites

[38]Bootstrapping Unsupervised Bilingual Lexicon Induction

G. Kondrak, B. Hauer, G. Nicolai - 2017

1 paper in library cites

[39]Compiling Bilingual Lexicon Entries From a Non-Parallel English-Chinese Corpus

Pascale Fung - 1995

1 paper in library cites

[40]Deciphering Related Languages

N. Pourdamghani, K. Knight - 2017

1 paper in library cites

[41]Earth mover's Distance Minimization for Unsupervised Bilingual Lexicon induction

Mingchuan Zhang, Yibo Liu, H. Luan, Maosong Sun - 2017

1 paper in library cites

[42]Hubs in Space: Popular Nearest Neighbors in High-Dimensional Data

M. Radovanovic, A. Nanopoulos, M. Ivanovic - 2010

1 paper in library cites

[43]Identifying Word Translations in Non-Parallel Texts

R. Rapp - 1995

1 paper in library cites

[44]Improving Zero-Shot Learning by Mitigating the Hubness Problem

G. D'souza, N. Vasic, A. R. Soriano, Yann Dauphin, M. Florescu - 2015

1 paper in library cites

[45]Inducing Crosslingual Distributed Representations of Words

A. Klementiev, I. Titov, B. Bhattarai - 2012

1 paper in library cites

[46]Inducing Translation Lexicons via Diverse Similarity Measures and Bridge Languages

C. Schafer, D. Yarowsky - 2002

1 paper in library cites

[47]Learning Crosslingual Word Embeddings Without bilingual corpora

L. Duong, H. Kanayama, T. Ma, S. Bird, T. Cohn - 2016

1 paper in library cites

[48]Nasari: Integrating Explicit Knowledge and Corpus Statistics for a Multilingual Representation of Concepts and Entities

J. C. Collados, M. T. Pilehvar, R. Navigli - 2016

1 paper in library cites

[49]Parseval Networks: Improving Robustness to Adversarial Examples

M. Cisse, Piotr Bojanowski, E. Grave, Yann Dauphin, Nicolas Usunier - 2017

1 paper in library cites

[50]Self-Tuning Spectral Clustering

L. Z. Manor, Pietro Perona - 2005

1 paper in library cites

[51]Supervised Bilingual Lexicon Induction With Multiple Monolingual Signals

A. Irvine, Chris Callison Burch - 2013

1 paper in library cites

[52]The Wacky Wide Web: A Collection of Very Large Linguistically Processed Web-Crawled Corpora

M. Baroni, S. Bernardini, A. Ferraresi, E. Zanchetta - 2009

1 paper in library cites

Cited by

3

papers in your library

Cites

16

papers in your library

Read

on November 3, 2025

I really really liked the methodology, but it is a bit hard to read given the math jargon they use. Also, the task (word translation) is very boring and I think lacks practical usages

Tags

Paper Aliases

No aliases