Papperoni

2019

A Survey of Reinforcement Learning Informed by Natural Language

Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, Tim Rocktaschel

Open PDF Google Scholar

citations

Cite Score

17

AI summary

This paper surveys the state of Reinforcement Learning (RL) informed by natural language, highlighting existing works in instruction following, text games, and learning from textual domain knowledge, and advocating for new environments and NLP techniques.

Main Contributions

Surveys the state of the field of Reinforcement Learning informed by Natural Language.
Categorizes existing work into language-conditional RL and language-assisted RL.
Reviews instruction following, rewards from instructions, and language in observation and action space.
Discusses the use of language for communicating domain knowledge and structuring policies.
Advocates for developing new environments that utilize domain knowledge in natural language and increased use of NLP methods like pre-trained language models.

Abstract

To be successful in real-world tasks, Reinforcement Learning (RL) needs to exploit the compositional, relational, and hierarchical structure of the world, and learn to transfer it to the task at hand. Recent advances in representation learning for language make it possible to build models that acquire world knowledge from text corpora and integrate this knowledge into downstream decision making problems. We thus argue that the time is right to investigate a tight integration of natural language understanding into RL in particular. We survey the state of the field, including work on instruction following, text games, and learning from textual domain knowledge. Fi- nally, we call for the development of new environments as well as further investigation into the po- tential uses of recent Natural Language Processing (NLP) techniques for such tasks.

Citation Graph

Loading graph...

References [86]

Sort:

Filter:

[1]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]Language Models Are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, D. Luan, Dario Amodei, Ilya Sutskever - 2019

27 papers in library cite

Amazing! Tons of important contributions. I think they could have explained the models a bit better, and I think this is where OpenAI starts to become evil (and not open)

[4]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[5]Universal Language Model Fine-Tuning for Text Classification

J. Howard, Sebastian Ruder - 2018

14 papers in library cite

Amazing! Bridging the gap between pre-training/finetuning in CV vs. NLP, plus giving amazing resuts!

[6]Wizard of Wikipedia: Knowledge-Powered Conversational Agents

E. Dinan, S. Roller, K. Shuster, A. Fan, Michael Auli, Jason Weston - 2019

4 papers in library cite

This could be the original RAG. It's soooo close to the original thing!

[7]SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Yejin Choi - 2018

5 papers in library cite

Very nice methodology with the adversarial stuff! I think not too relevant though.

[8]Reinforcement Learning: An Introduction

Richard S. Sutton, A. Barto - 1998

5 papers in library cite

Book

[9]Indexing by Latent Semantic Analysis

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman - 1990

12 papers in library cite

LSA paper

[10]Vqa: Visual Question Answering

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, D. Parikh - 2015

6 papers in library cite

New task?

[11]Generative Adversarial Imitation Learning

J. Ho, Stefano Ermon - 2016

3 papers in library cite

[12]Maximum Entropy Inverse Reinforcement Learning

B. D. Ziebart, A. Maas, J. A. Bagnell, A. K. Dey - 2008

1 paper in library cites

[13]Devise: A Deep Visual-Semantic Embedding Model

Andrea Frome, G. S. Corrado, J. Shlens, Samy Bengio, Jeffrey Dean, Tomas Mikolov, Marc'aurelio Ranzato - 2013

4 papers in library cite

Identify objects

[14]Reading Wikipedia to Answer Open-Domain Questions

Deli Chen, Adam Fisch, Jason Weston, Antoine Bordes - 2017

10 papers in library cite

Open Domain QA with wikipedia

[15]An Algorithmic Perspective on Imitation Learning

T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters - 2018

1 paper in library cites

[16]Language Understanding for Text-Based Games Using Deep Reinforcement Learning

K. Narasimhan, T. Kulkarni, R. Barzilay - 2015

2 papers in library cite

Application on games!

[17]Learning to Understand Goal Specifications by Modelling Reward

D. Bahdanau, F. Hill, Jan Leike, E. Hughes, P. Kohli, Edward Grefenstette - 2019

4 papers in library cite

[18]What Do You Learn From Context? Probing for Sentence Structure in Contextualized Word Representations

I. Tenney, P. Xia, Berlin Chen, A. Wang, A. Poliak, R. T. Mccoy, N. Kim, B. V. Durme, S. Bowman, Dipanjan Das, Ellie Pavlick - 2019

4 papers in library cite

Analysis of how transformers learn phrase structure

[19]A Brief Survey of Deep Reinforcement Learning

K. Arulkumaran, M. P. Deisenroth, Miles Brundage, A. A. Bharath - 2017

1 paper in library cites

[20]Mastering the Game of Go Without Human Knowledge

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Yanru Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, Demis Hassabis - 2017

7 papers in library cite

[21]Assessing bert's Syntactic Abilities

Y. Goldberg - 2019

4 papers in library cite

[22]Dissecting Contextual Word Embeddings: Architecture and Representation

M. E. Peters, M. Neumann, Luke Zettlemoyer, W. T. Yih - 2018

4 papers in library cite

[23]Learning Language Games Through Interaction

S. I. Wang, Percy Liang, Christopher D. Manning - 2016

3 papers in library cite

[24]Clevr: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

J. Johnson, B. Hariharan, Laurens Van Der Maaten, Li Fei Fei, C. L. Zitnick, Ross Girshick - 2017

2 papers in library cite

[25]Guiding a Reinforcement Learner With Natural Language Advice: Initial Results in Robocup Soccer

G. Kuhlmann, P. Stone, R. Mooney, J. Shavlik - 2004

2 papers in library cite

[26]Human Behavior and the Principle of Least Effort

G. K. Zipf - 1949

2 papers in library cite

[27]Learning to Interpret Natural Language Navigation Instructions From Observations

Deli Chen, R. Mooney - 2011

2 papers in library cite

[28]Learning to Win by Reading Manuals in a Monte-Carlo Framework

S. R. K. Branavan, D. Silver, R. Barzilay - 2011

2 papers in library cite

[29]Learning With Latent Language

Jacob Andreas, Dan Klein, Sergey Levine - 2018

2 papers in library cite

[30]Recent Advances in Hierarchical Reinforcement Learning

A. G. Barto, S. Mahadevan - 2003

2 papers in library cite

[31]Temporal Difference Learning and TD-Gammon

Gerald Tesauro - 1995

2 papers in library cite

[32]Textworld: A Learning Environment for Text-Based Games

M. A. Cote, A. Kadar, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, M. J. Hausknecht, L. E. Asri, M. Adada, W. Tay, A. Trischler - 2018

2 papers in library cite

[33]Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

H. Chen, A. Suhr, D. K. Misra, N. Snavely, Y. Artzi - 2018

2 papers in library cite

[34]A Survey on Dialogue Systems: Recent Advances and New Frontiers

H. Chen, Xiaodong Liu, D. Yin, Jie Tang - 2017

1 paper in library cites

[35]A Synopsis of Linguistic Theory

J. R. Firth - 1957

1 paper in library cites

[36]Alignment-Based Compositional Semantics for Instruction Following

Jacob Andreas, Dan Klein - 2015

1 paper in library cites

[37]Apprenticeship Learning About Multiple Intentions

M. Babes, V. Marivate, K. Subramanian, M. L. Littman - 2011

1 paper in library cites

[38]BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning

M. C. Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, Yoshua Bengio - 2019

1 paper in library cites

[39]Being-in-the-World

M. A. Depristo, R. Zubek - 2001

1 paper in library cites

[40]CHALET: cornell house agent Learning Environment

C. Yan, D. K. Misra, A. Bennett, A. Walsman, Yonatan Bisk, Y. Artzi - 2018

1 paper in library cites

[41]Cognitive Effects of Language on Human Navigation

A. Shusterman, S. A. Lee, E. Spelke - 2011

1 paper in library cites

[42]Core Knowledge

E. Spelke, K. D. Kinzler - 2007

1 paper in library cites

[43]Counting to Explore and Generalize in Text-Based Games

X. Yuan, M. A. Cote, A. Sordoni, R. Laroche, R. T. D. Combes, M. J. Hausknecht, A. Trischler - 2018

1 paper in library cites

[44]Deep Reinforcement Learning for General Video Game AI

R. R. Torrado, P. Bontrager, J. Togelius, Joseph Liu, D. P. Liebana - 2018

1 paper in library cites

[45]ELF: An Extensive, Lightweight and Flexible Research Platform for Real-Time Strategy Games

Yuandong Tian, Q. Gong, W. Shang, Yonghui Wu, C. L. Zitnick - 2017

1 paper in library cites

[46]Embodied Question Answering

A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, D. Batra - 2018

1 paper in library cites

[47]Flipdial: A Generative Model for Two-Way Visual Dialogue

D. Massiceti, N. Siddharth, P. K. Dokania, P. H. Torr - 2018

1 paper in library cites

[48]From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

J. Fu, A. Korattikara, Sergey Levine, S. Guadarrama - 2019

1 paper in library cites

[49]Gated-Attention Architectures for Task-Oriented Language Grounding

D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, Ruslan Salakhutdinov - 2018

1 paper in library cites

[50]Grounded Language Learning in a Simulated 3d World

K. M. Hermann, F. Hill, S. Green, Feng Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki, M. Jaderberg, D. Teplyashin, M. Wainwright, C. Apps, Demis Hassabis, Phil Blunsom - 2017

1 paper in library cites

[51]Grounding English Commands to Reward Functions

J. Macglashan, M. B. Vroman, M. Desjardins, M. L. Littman, S. Muresan, S. Squire, S. Tellex, D. Arumugam, L. Yang - 2015

1 paper in library cites

[52]Grounding Language for Transfer in Deep Reinforcement Learning

K. Narasimhan, R. Barzilay, T. Jaakkola - 2018

1 paper in library cites

[53]Handbook of Intelligent Control: Neural, Fuzzy, and Adaptative Approaches

D. A. White, D. A. Sofge - 1992

1 paper in library cites

[54]Hierarchical and Interpretable Skill Acquisition in Multi-Task Reinforcement Learning

T. Shu, Caiming Xiong, Richard Socher - 2018

1 paper in library cites

[55]Hierarchical Decision Making by Generating and Following Natural Language Instructions

Han Hu, D. Yarats, Q. Gong, Yuandong Tian, Martha Lewis - 2019

1 paper in library cites

[56]Human Learning in Atari

P. Tsividis, T. Pouncy, J. L. L. Xu, Joshua B. Tenenbaum, S. J. Gershman - 2017

1 paper in library cites

[57]Interactive Grounded Language Acquisition and Generalization in a 2d World

H. Yu, Haowei Zhang, Weixin Xu - 2018

1 paper in library cites

[58]Iqa: Visual Question Answering in Interactive Environments

D. Gordon, A. Kembhavi, M. Rastegari, Joseph Redmon, D. Fox, Ali Farhadi - 2018

1 paper in library cites

[59]Learning to Generalize From Sparse and Underspecified Rewards

R. Agarwal, C. Liang, Dale Schuurmans, M. Norouzi - 2019

1 paper in library cites

[60]Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences

H. Mei, Mohit Bansal, M. R. Walter - 2016

1 paper in library cites

[61]Mapping Instructions and Visual Observations to Actions With Reinforcement Learning

Diganta Misra, John Langford, Y. Artzi - 2017

1 paper in library cites

[62]Modular Multitask Reinforcement Learning With Policy Sketches

Jacob Andreas, Dan Klein, Sergey Levine - 2017

1 paper in library cites

[63]Natural Language Communication With Robots

Yonatan Bisk, Deniz Yuret, D. Marcu - 2016

1 paper in library cites

[64]Neural Modular Control for Embodied Question Answering

A. Das, G. Gkioxari, S. Lee, D. Parikh, D. Batra - 2018

1 paper in library cites

[65]Neural Module Networks

Jacob Andreas, M. Rohrbach, Trevor Darrell, Dan Klein - 2016

1 paper in library cites

[66]Open Information Extraction From the Web

M. Banko, Michael J. Cafarella, S. Soderland, M. Broadhead, Oren Etzioni - 2007

1 paper in library cites

[67]Optimizing Dialogue Management With Reinforcement Learning: Experiments With the Njfun System

Shivalika Singh, D. Litman, M. Kearns, Mitch Walker - 2002

1 paper in library cites

[68]Pragmatics: An Introduction

J. Mey - 1993

1 paper in library cites

[69]Question Answering Systems: Survey and Trends

A. Bouziane, D. Bouchiha, N. Doumi, M. Malki - 2015

1 paper in library cites

[70]Reading Between the Lines: Learning to Map High-Level Instructions to Commands

S. R. K. Branavan, L. S. Zettlemoyer, R. Barzilay - 2010

1 paper in library cites

[71]Reading to Learn: Constructing Features From Semantic Abstracts

J. Eisenstein, J. Clarke, D. Goldwasser, Dan Roth - 2009

1 paper in library cites

[72]Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xinpeng Wang, Q. Huang, A. Celikyilmaz, Jianfeng Gao, D. Shen, Y. F. Wang, W. Y. Wang, Li Zhang - 2018

1 paper in library cites

[73]Representation Learning for Grounded Spatial Reasoning

M. Janner, K. Narasimhan, R. Barzilay - 2018

1 paper in library cites

[74]Text-Based Adventures of the Golovin AI Agent

B. Kostka, J. Kwiecieli, J. Kowalski, P. Rychlikowski - 2017

1 paper in library cites

[75]The Development of Categorization in the Second Year and Its Relation to Other Cognitive and Linguistic Developments

A. Gopnik, A. Meltzoff - 1987

1 paper in library cites

[76]The Malmo Platform for Artificial Intelligence Experimentation

M. J. Johnson, K. Hofmann, T. Hutton, D. Bignell - 2016

1 paper in library cites

[77]Toward Understanding Natural Language Directions

T. Kollar, S. Tellex, D. Roy, N. Roy - 2010

1 paper in library cites

[78]Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation

S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, N. Roy - 2011

1 paper in library cites

[79]Using Natural Language for Reward Shaping in Reinforcement Learning

P. Goyal, S. Niekum, R. J. Mooney - 2019

1 paper in library cites

[80]Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

P. Anderson, Q. Wu, D. Teney, J. Bruce, M. J. Johnson, N. Sunderhauf, I. Reid, S. Gould, A. V. D. Hengel - 2018

1 paper in library cites

[81]Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions

M. Macmahon, B. Stankiewicz, B. Kuipers - 2006

1 paper in library cites

[82]Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

Y. Artzi, Luke Zettlemoyer - 2013

1 paper in library cites

[83]What Can You Do With a Rock? Affordance Extraction via Word Embeddings

N. Fulda, D. Ricks, B. Murdoch, D. Wingate - 2017

1 paper in library cites

[84]Zero-Shot Learning Through Cross-Modal Transfer

Richard Socher, M. Ganjoo, Christopher D. Manning, Andrew Y. Ng - 2013

1 paper in library cites

[85]Zero-Shot Task Generalization With Multi-Task Deep Reinforcement Learning

J. Oh, S. P. Singh, Honglak Lee, P. Kohli - 2017

1 paper in library cites

Infocom - 1980

1 paper in library cites

Cited by

3

papers in your library

Cites

19

papers in your library

Read

on January 11, 2026

It's a good overview and I like that it gave me context on what's happening, but a bit boring to read.

Tags

Paper Aliases

No aliases