Papperoni

2018

Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer

Open PDF Google Scholar

citations

Cite Score

90

AI summary

This paper introduces ELMo (Embeddings from Language Models), a novel deep contextualized word representation using bidirectional LSTMs, pre-trained on a large corpus, that significantly improves state-of-the-art results on six NLP tasks.

Main Contributions

Introduces ELMo, a new type of deep contextualized word representation.
ELMo word vectors are learned functions of the internal states of a deep bidirectional language model (biLM).
Demonstrates that ELMo representations can be easily added to existing models.
Achieves significant improvements on six challenging NLP problems: question answering, textual entailment, sentiment analysis, semantic role labeling, coreference resolution, and named entity extraction.
Shows that exposing the deep internals of the pre-trained network is crucial.

Abstract

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.

Citation Graph

Loading graph...

References [61]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[4]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[5]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[6]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[7]Enriching Word Vectors With Subword Information

Tomas Mikolov - 2017

7 papers in library cite

It's just word2vec with ngrams instead of full words. Unsurprising, but important nonetheless

[8]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[9]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[10]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[11]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[12]On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Kyunghyun Cho, B. V. Merrienboer, D. Bahdanau, Yoshua Bengio - 2014

9 papers in library cite

This just builds upon other work, comparing and analyzing previous architectures. This faces a specific problem of the time: long sentences. Not too relevant today.

[13]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[14]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[15]Training Very Deep Networks

R. K. Srivastava, K. Greff, Jürgen Schmidhuber - 2015

6 papers in library cite

The idea of using the same concepts of LSTM to feedforward nets is so simple yet it took so long! Very cool idea but the results are underwhelming (otherwise would be a 5!)

[16]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[17]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[18]A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal - 2015

9 papers in library cite

The methodology is very simple and effective, but maybe it was too simple for a paper, so they put A TON of math on top of it... Unnecessary really.

[19]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[20]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[21]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[22]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[23]Semi-Supervised Sequence Tagging With Bidirectional Language Models

M. E. Peters, W. Ammar, C. Bhagavatula, Russell Power - 2017

5 papers in library cite

Nothing too different from other LSTM + Attention papers

[24]context2vec: Learning Generic Context Embedding With Bidirectional LSTM

O. Melamud, J. Goldberger, Ido Dagan - 2016

5 papers in library cite

I liked the methodology a lot but the paper reads like an MSc course project.

[25]Simple and Effective Multi-Paragraph Reading Comprehension

C. Clark, Matt Gardner - 2017

7 papers in library cite

Very nice paper! I think it's a stretch to call it "simple", but the paper is very well written and easy to follow.

[26]Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, Victor Zhong, R. Paulus, Richard Socher - 2015

9 papers in library cite

[27]Enhanced LSTM for Natural Language Inference

Qinlang Chen, X. Zhu, Z. H. Ling, S. Wei, H. Jiang, D. Inkpen - 2017

5 papers in library cite

SotA NLI

[28]Regularizing and Optimizing 1stm Language Models

S. Merity, Nitish Shirish Keskar, Richard Socher - 2017

6 papers in library cite

SotA LM in 2017

[29]Gated Self-Matching Networks for Reading Comprehension and Question Answering

Wenyi Wang, N. Yang, F. Wei, B. Chang, M. Zhou - 2017

5 papers in library cite

QA Model

[30]On the State of the Art of Evaluation in Neural Language Models

G. Melis, C. Dyer, Phil Blunsom - 2018

6 papers in library cite

SotA LM in 2017

[31]What Do Neural Machine Translation Models Learn About Morphology?

Yonatan Belinkov, N. Durrani, F. Dalvi, H. Sajjad, J. R. Glass - 2017

2 papers in library cite

They perform an "intrinsic evaluation" of words to understand what they represent

[32]Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison

A. Raganato, J. C. Collados, R. Navigli - 2017

2 papers in library cite

Framework for word sense disambiguation

[33]Improving Sequence to Sequence Learning With Unlabeled Data

P. Ramachandran, P. Liu, Quoc Le - 2017

1 paper in library cites

Unsupervised pre-training (Unsupervised Pretraining for Sequence to Sequence Learning)

[34]Neural Sequence Learning Models for Word Sense Disambiguation

A. Raganato, C. D. Bovi, R. Navigli - 2017

1 paper in library cites

Disambiguation of words

[35]Character-Aware Neural Language Models

Yoon Kim, Yacine Jernite, D. Sontag, Alexander M. Rush - 2016

7 papers in library cite

[36]Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

J. Lafferty, Andrew Mccallum, F. C. Pereira - 2001

6 papers in library cite

[37]A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

K. Hashimoto, Caiming Xiong, Y. Tsuruoka, Richard Socher - 2016

5 papers in library cite

[38]Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation

W. Ling, C. Dyer, A. W. Black, I. Trancoso, R. Fermandez, S. Amir, L. Marujo, T. Luis - 2015

5 papers in library cite

[39]An Empirical Exploration of Recurrent Network Architectures

R. Jozefowicz, Wojciech Zaremba, Ilya Sutskever - 2015

4 papers in library cite

[40]Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

E. F. T. K. Sang, F. D. Meulder - 2003

4 papers in library cite

[41]Neural Architectures for Named Entity Recognition

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer - 2016

4 papers in library cite

[42]Neural Tree Indexers for Text Understanding

H. Yu, T. Munkhdalai - 2017

4 papers in library cite

[43]Deep Multi-Task Learning With Low Level Tasks Supervised at Lower Layers

A. Sogaard, Y. Goldberg - 2016

3 papers in library cite

[44]End-to-End Learning of Semantic Role Labeling Using Recurrent Neural Networks

Jingren Zhou, Weixin Xu - 2015

3 papers in library cite

[45]End-to-End Neural Coreference Resolution

K. Lee, Luheng He, Martha Lewis, L. S. Zettlemoyer - 2017

3 papers in library cite

[46]End-to-End Sequence Labeling via Bi-Directional LSTM-CNNS-CRF

X. Ma, Eduard Hovy - 2016

3 papers in library cite

[47]Text Classification Improved by Integrating Bidirectional LSTM With Two-Dimensional Max Pooling

P. Zhou, Z. Qi, S. Zheng, Jiacheng Xu, H. Bao, B. Xu - 2016

3 papers in library cite

[48]The Proposition Bank: An Annotated Corpus of Semantic Roles

M. Palmer, P. Kingsbury, D. Gildea - 2005

3 papers in library cite

[49]CHARAGRAM: Embedding Words and Sentences via Character N-Grams

J. Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu - 2016

2 papers in library cite

[50]Conll-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in Ontonotes

S. Pradhan, A. Moschitti, N. Xue, O. Uryupina, Y. Z. Zhang - 2012

2 papers in library cite

[51]Deep Semantic Role Labeling: What Works and What's Next

Luheng He, K. Lee, Martha Lewis, L. S. Zettlemoyer - 2017

2 papers in library cite

[52]Named Entity Recognition With Bidirectional LSTM-CNNs

J. Chiu, E. Nichols - 2016

2 papers in library cite

[53]Natural Language Inference Over Interaction Space

Y. Gong, H. Luo, J. Zhang - 2018

2 papers in library cite

[54]Stochastic Answer Networks for Machine Reading Comprehension

Xiaodong Liu, Y. Shen, K. Duh, Jianfeng Gao - 2017

2 papers in library cite

[55]Deep Reinforcement Learning for Mention-Ranking Coreference Models

K. Clark, Christopher D. Manning - 2016

1 paper in library cites

[56]Easy Victories and Uphill Battles in Coreference Resolution

G. Durrett, Dan Klein - 2013

1 paper in library cites

[57]Efficient Non-Parametric Estimation of Multiple Embeddings Per Word in Vector Space

Arvind Neelakantan, J. Shankar, A. Passos, Andrew Mccallum - 2014

1 paper in library cites

[58]Embeddings for Word Sense Disambiguation: An Evaluation Study

I. Iacobacci, M. T. Pilehvar, R. Navigli - 2016

1 paper in library cites

[59]Learning Global Features for Coreference Resolution

Sam Wiseman, Alexander M. Rush, Stuart M. Shieber - 2016

1 paper in library cites

[60]Towards Robust Linguistic Analysis Using Ontonotes

S. Pradhan, A. Moschitti, N. Xue, H. Ng, A. Bjorkelund, O. Uryupina, Y. Z. Zhang, Z. Zhong - 2013

1 paper in library cites

[61]Using a Semantic Concordance for Sense Identification

George A. Miller, M. Chodorow, S. Landes, C. Leacock, R. G. Thomas - 1994

1 paper in library cites

Cited by

27

papers in your library

Cites

34

papers in your library

Read

on August 7, 2025

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

Tags

Paper Aliases

No aliases