Papperoni

2017

Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher

Open PDF Google Scholar

citations

Cite Score

42

AI summary

This paper introduces a method to contextualize word vectors (CoVe) using a deep LSTM encoder from a machine translation model, demonstrating improved performance on sentiment analysis, question classification, entailment, and question answering tasks, achieving state-of-the-art results on fine-grained sentiment analysis and entailment.

Main Contributions

Introduces context vectors (CoVe) from a MT-LSTM to improve NLP task performance.
Demonstrates that CoVe improves performance over using unsupervised word and character vectors on various NLP tasks.
Achieves state-of-the-art performance on fine-grained sentiment analysis and entailment tasks using CoVe.
Shows that the quantity of MT training data positively correlates with downstream task performance.
Presents a general biattentive classification network (BCN) for testing CoVe transferability.

Abstract

Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art.

Citation Graph

Loading graph...

References [71]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[4]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[5]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[6]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[7]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[8]Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Ross Girshick, J. Donahue, Trevor Darrell, Jitendra Malik - 2014

18 papers in library cite

Good results, beat overfeat, used pretraining for improving performance. Only issue is that the paper is overly long...

[9]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[10]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[11]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[12]Effective Approaches to Attention-Based Neural Machine Translation

T. Luong, H. Pham, Christopher D. Manning - 2015

15 papers in library cite

Good paper, but very derivative. Attention methods start getting very complicated... I understand why Transformers took over TBH

[13]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[14]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[15]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[16]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[17]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[18]Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. N. D. Santos, C. G. Gulcehre, Bing Xiang - 2016

10 papers in library cite

Cool that they use pointer switching, and they introduce the CNN/daily mail dataset for summarization, which is a nice insight.

[19]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[20]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[21]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[22]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[23]A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit - 2016

11 papers in library cite

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

[24]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[25]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[26]Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever - 2017

8 papers in library cite

Very impressive results. We can see that they were onto something (later became GPT2)

[27]Unsupervised Pretraining for Sequence to Sequence Learning

P. Ramachandran, P. J. Liu, Quoc V. Le - 2017

9 papers in library cite

It's alright, but it's the same Seq2Seq thing with pretraining

[28]Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures

Alex Graves, Jürgen Schmidhuber - 2005

14 papers in library cite

Very nice paper! Simple, no bullshit. Just "hey, we have LSTM and we have BRNN, let's try to join it"

[29]OpenNMT: Open-Source Toolkit for Neural Machine Translation

G. Klein, Yoon Kim, Y. Deng, J. Senellart, A. Rush - 2017

4 papers in library cite

Framework that I never heard before...

[30]Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

A. Fukui, D. H. Park, Diyi Yang, A. Rohrbach, Trevor Darrell, M. Rohrbach - 2016

2 papers in library cite

Visual QA

[31]Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

A. Kumar, O. Irsoy, P. Ondruska, M. Iyyer, J. Bradbury, I. Gulrajani, Victor Zhong, R. Paulus, Richard Socher - 2015

9 papers in library cite

[32]Gated Self-Matching Networks for Reading Comprehension and Question Answering

Wenyi Wang, N. Yang, F. Wei, B. Chang, M. Zhou - 2017

5 papers in library cite

QA Model

[33]Towards Universal Paraphrastic Sentence Embeddings

J. Wieting, Mohit Bansal, Kevin Gimpel, K. A. Livescu, K. Karen - 2015

4 papers in library cite

[34]Machine Comprehension Using Match-LSTM and Answer Pointer

Shijie Wang, J. J. Jiang - 2017

6 papers in library cite

Answer pointer

[35]Moses: Open Source Toolkit for Statistical Machine Translation

P. Koehn, H. Hoang, Alexandra Birch, Chris Callison Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, E. Herbst - 2007

8 papers in library cite

[36]A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

K. Hashimoto, Caiming Xiong, Y. Tsuruoka, Richard Socher - 2016

5 papers in library cite

[37]Dynamic Memory Networks for Visual and Textual Question Answering

Caiming Xiong, S. Merity, Richard Socher - 2016

5 papers in library cite

[38]Grounded Compositional Semantics for Finding and Describing Images With Sentences

Richard Socher, Quoc Le, C. Manning, A. Ng - 2014

5 papers in library cite

[39]The trec-8 Question Answering Track Evaluation

E. M. Voorhees, D. M. Tice - 1999

5 papers in library cite

[40]Adversarial Training Methods for Semi-Supervised Text Classification

T. Miyato, A. M. Dai, I. Goodfellow - 2016

4 papers in library cite

[41]Neural Semantic Encoders

H. Yu, T. Munkhdalai - 2017

4 papers in library cite

[42]Neural Tree Indexers for Text Understanding

H. Yu, T. Munkhdalai - 2017

4 papers in library cite

[43]Question Answering Through Transfer Learning From Large Fine-Grained Supervision Data

S. Min, M. J. Seo, Hananneh Hajishirzi - 2017

4 papers in library cite

[44]SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

E. Agirre, C. Banea, C. Cardie, D. M. Cer, M. T. Diab, A. G. Agirre, W. Guo, R. Mihalcea, G. Rigau, J. Wiebe - 2014

4 papers in library cite

[45]Discriminative Neural Sentence Modeling by Tree-Based Convolution

L. Mou, H. Peng, G. Li, Yiheng Xu, Li Zhang, Z. Jin - 2015

3 papers in library cite

[46]Dynamic Coattention Networks for Question Answering

Caiming Xiong, Victor Zhong, Richard Socher - 2017

3 papers in library cite

[47]End-to-End Reading Comprehension With Dynamic Answer Chunk Ranking

Y. Yu, Wenxuan Zhang, K. S. Hasan, M. Yu, Bing Xiang, B. Zhou - 2016

3 papers in library cite

[48]From Symbolic to Sub-Symbolic Information in Question Classification

J. P. C. G. D. Silva, L. Coheur, A. C. Mendes, A. Wichert - 2011

3 papers in library cite

[49]Text Classification Improved by Integrating Bidirectional LSTM With Two-Dimensional Max Pooling

P. Zhou, Z. Qi, S. Zheng, Jiacheng Xu, H. Bao, B. Xu - 2016

3 papers in library cite

[50]Topicrnn: A Recurrent Neural Network With Long-Range Semantic Dependency

A. Dieng, A. B., Caitlin Wang, Jianfeng Gao, J. A. Paisley, J. John - 2016

3 papers in library cite

[51]Adapting Visual Category Models to New Domains

K. Saenko, B. Kulis, M. Fritz, Trevor Darrell - 2010

2 papers in library cite

[52]Deep Learning With Dynamic Computation Graphs

M. Looks, M. Herreshoff, M. Hutchins, D. A. Norvig, P. Peter - 2017

2 papers in library cite

[53]Recursive Neural Networks Can Learn Logical Semantics

Samuel R. Bowman, Christopher Potts, Christopher D. Manning - 2015

2 papers in library cite

[54]Supervised and Semi-Supervised Text Categorization Using LSTM for Region Embeddings

R. Johnson, Tong Zhang - 2016

2 papers in library cite

[55]The iwslt 2016 Evaluation Campaign

M. Cettolo, J. Niehues, S. Stuker, L. Bentivogli, R. Cattoni, M. J. Federico - 2016

2 papers in library cite

[56]A Neural Architecture Mimicking Humans End-to-End for Natural Language Inference

B. Paria, K. M. Annervaz, A. Dukkipati, A. Chatterjee, S. Podder - 2016

1 paper in library cites

[57]A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

L. Specia, S. Frank, K. Sima'an, D. Elliott - 2016

1 paper in library cites

[58]Dependency Sensitive Convolutional Neural Networks for Modeling Sentences and Documents

Robert Zhang, Honglak Lee, D. R. Radev - 2016

1 paper in library cites

[59]Encoding Syntactic Knowledge in Neural Networks for Sentiment Classification

M. Huang, Q. Qian, X. Zhu - 2017

1 paper in library cites

[60]End-to-End Multi-View Networks for Text Classification

H. Guo, C. Cherry, Jianlin Su - 2017

1 paper in library cites

[61]Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference

Qinlang Chen, X. D. Zhu, Z. H. Ling, S. Wei, H. Jiang - 2016

1 paper in library cites

[62]Hedged Deep Tracking

Y. Qi, S. Zhang, Lianhui Qin, H. Yao, Q. Huang, J. Lim, M. H. Yang - 2016

1 paper in library cites

[63]Heterogeneous Transfer Learning for Image Classification

Yuxuan Zhu, Yanru Chen, Z. L. Lu, Sinno Jialin Pan, G. R. Xue, Y. Yu, Qiang Yang - 2011

1 paper in library cites

[64]High Accuracy Rule Based Question Classification Using Question Syntax and Semantics

H. T. Madabushi, M. Lee - 2016

1 paper in library cites

[65]Improving Question Classification by Feature Extraction and Selection

N. V. Tu, L. A. Cuong - 2016

1 paper in library cites

[66]Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning

J. Lu, Caiming Xiong, D. Parikh, Richard Socher - 2016

1 paper in library cites

[67]Language to Logical Form With Neural Attention

L. Dong, Mirella Lapata - 2016

1 paper in library cites

[68]Learning Question Classifiers: The Role of Semantic Information

Xiang Lisa Li, Dan Roth - 2006

1 paper in library cites

[69]Question Classification by Weighted Combination of Lexical, Syntactic and Semantic Features

B. Loni, G. V. Tulder, P. Wiggers, D. M. J. Tax, M. Loog - 2011

1 paper in library cites

[70]Reading and Thinking: Re-Read LSTM unit for Textual Entailment Recognition

L. Sha, B. Chang, Zhifang Sui, Shanda Li - 2016

1 paper in library cites

[71]The Representational Geometry of Word Meanings Acquired by Neural Machine Translation Models

F. Hill, Kyunghyun Cho, S. Jean, Yoshua Bengio - 2017

1 paper in library cites

Cited by

14

papers in your library

Cites

34

papers in your library

Read

on October 23, 2025

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

Tags

Paper Aliases

No aliases