Papperoni

2019

Cross-Lingual Language Model Pretraining

G. Lample, Alexis Conneau

Open PDF Google Scholar

citations

Cite Score

51

AI summary

This paper introduces cross-lingual language models (XLMs) with unsupervised and supervised objectives, achieving state-of-the-art results on XNLI cross-lingual classification with a 4.9% gain, 34.3 BLEU on unsupervised WMT'16 German-English translation (9 BLEU improvement), and 38.5 BLEU on supervised WMT'16 Romanian-English translation (4 BLEU improvement).

Main Contributions

Introduces a new unsupervised method for learning cross-lingual representations using cross-lingual language modeling and investigates two monolingual pretraining objectives.
Introduces a new supervised learning objective that improves cross-lingual pretraining when parallel data is available.
Significantly outperforms the previous state of the art on cross-lingual classification, unsupervised machine translation and supervised machine translation.
Shows that cross-lingual language models can provide significant improvements on the perplexity of low-resource languages.
Makes code and pretrained models publicly available.

Abstract

Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding. In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.

Citation Graph

Loading graph...

References [45]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[4]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[5]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[6]Automatic Differentiation in PyTorch

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Zongyu Lin, A. Desmaison, L. Antiga, Adam Lerer - 2017

3 papers in library cite

Very boring and poorly explained. It looks like they said "we need to cash in some citations! quick, publish anything you have!!!"

[7]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[8]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[9]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[10]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[11]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[12]Bridging Nonlinearities and Stochastic Regularizers With Gaussian Error Linear Units

Dan Hendrycks, Kevin Gimpel - 2016

9 papers in library cite

Very understandable, and very nice! I don't think the justification is good, but hey, it works!

[13]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[14]Backpropagation Through Time: What It Does and How to Do It

P. Werbos - 1990

9 papers in library cite

Amazing tutorial! Very pragmatic. Explains very basic concepts and focus on implementation

[15]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[16]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[17]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[18]Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever - 2013

6 papers in library cite

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

[19]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[20]Word Translation Without Parallel Data

Alexis Conneau, G. Lample, Marc'aurelio Ranzato, L. Denoyer, Hervé Jégou - 2018

3 papers in library cite

I really really liked the methodology, but it is a bit hard to read given the math jargon they use. Also, the task (word translation) is very boring and I think lacks practical usages

[21]Unsupervised Neural Machine Translation

M. Artetxe, G. Labaka, E. Agirre, Kyunghyun Cho - 2017

4 papers in library cite

Very nice methodology! I am impressed that this is even possible. However, results are somewhat underwhelming...

[22]Character-Level Language Modeling With Deeper Self-Attention

R. A. Rfou, D. Choe, Noah Constant, M. Guo, Llion Jones - 2018

6 papers in library cite

It's good, and almost a 4, but I think it is a bit boring. Plus character LMs are not meta.

[23]Unsupervised Pretraining for Sequence to Sequence Learning

P. Ramachandran, P. J. Liu, Quoc V. Le - 2017

9 papers in library cite

It's alright, but it's the same Seq2Seq thing with pretraining

[24]Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

M. J. Johnson, M. Schuster, Quoc V. Le, M. Krikun, Yonghui Wu, Ziru Chen, N. Thorat, F. B. Viegas, M. Wattenberg, G. S. Corrado, M. Hughes, Jeffrey Dean - 2017

7 papers in library cite

Google's NMT system V2?

[25]Unsupervised Machine Translation Using monolingual corpora Only

G. Lample, L. Denoyer, Marc'aurelio Ranzato - 2017

4 papers in library cite

How?

[26]Senteval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela - 2018

5 papers in library cite

Sentence embedding eval toolkit?

[27]Offline Bilingual Word Vectors, Orthogonal Transformations and the Inverted Softmax

S. L. Smith, D. H. Turban, S. Hamblin, N. Y. Hammerla - 2017

4 papers in library cite

Inverted softmax

[28]Moses: Open Source Toolkit for Statistical Machine Translation

P. Koehn, H. Hoang, Alexandra Birch, Chris Callison Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, E. Herbst - 2007

8 papers in library cite

[29]Edinburgh neural Machine Translation Systems for WMT 16

R. Sennrich, B. Haddow, Alexandra Birch - 2016

5 papers in library cite

[30]Cloze Procedure: A New Tool for Measuring Readability

W. L. Taylor - 1953

4 papers in library cite

[31]Improving Vector Space Word Representations Using Multilingual Correlation

Manaal Faruqui, C. Dyer - 2014

2 papers in library cite

[32]Massively Multilingual Word Embeddings

W. Ammar, G. Mulcaire, Y. Tsvetkov, G. Lample, C. Dyer, Noah A. Smith - 2016

2 papers in library cite

[33]Multilingual Models for Compositional Distributional Semantics

K. M. Hermann, Phil Blunsom - 2014

2 papers in library cite

[34]Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

C. Xing, D. Wang, C. L. Liu, Yutong Lin - 2015

2 papers in library cite

[35]Parallel Data, Tools and Interfaces in Opus

J. Tiedemann - 2012

2 papers in library cite

[36]Semeval-2017 Task 2: Multilingual and Cross-Lingual Semantic Word Similarity

J. C. Collados, M. T. Pilehvar, N. Collier, R. Navigli - 2017

2 papers in library cite

[37]Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

M. Artetxe, Holger Schwenk - 2018

1 paper in library cites

[38]Optimizing Chinese Word Segmentation for Machine Translation Performance

P. C. Chang, M. Galley, Christopher D. Manning - 2008

1 paper in library cites

[39]Phrase-Based & neural Unsupervised Machine Translation

G. Lample, M. Ott, Alexis Conneau, L. Denoyer, Marc'aurelio Ranzato - 2018

1 paper in library cites

[40]The iit bombay english-hindi parallel corpus

K. Anoop, M. Pratik, B. Pushpak - 2018

1 paper in library cites

[41]The united nations parallel corpus v1. 0

M. Ziemski, M. J. Dowmunt, B. Pouliquen - 2016

1 paper in library cites

[42]Transformer-XL: Language Modeling With Longer-Term Dependency

Z. Dai, Zhilin Yang, Yining Yang, W. W. Cohen, J. Carbonell, Quoc V. Le, Ruslan Salakhutdinov - 2019

1 paper in library cites

[43]Unsupervised Cross-Lingual Word Embedding by Multilingual Neural Language Models

T. Wada, T. Iwata - 2018

1 paper in library cites

[44]Xnli: Evaluating Cross-Lingual Sentence Representations

Alexis Conneau, R. Rinott, G. Lample, A. Williams, Samuel R. Bowman, Holger Schwenk, Veselin Stoyanov - 2018

1 paper in library cites

[45]Zero-Shot Cross-Lingual Classification Using Multilingual Neural Machine Translation

A. Eriguchi, M. J. Johnson, O. Firat, H. Kazawa, W. Macherey - 2018

1 paper in library cites

Cited by

5

papers in your library

Cites

27

papers in your library

Read

on November 17, 2025

I like how they learn cross-lingual stuff via vocabulary sharing. it's probably the start of the multi-language LMs.

Tags

Paper Aliases

No aliases