Papperoni

2009

Flexible Shaping: How Learning in Small Steps Helps

K. A. Krueger, Peter Dayan

Open PDF Google Scholar

citations

Cite Score

14

AI summary

This paper introduces a LSTM network to model flexible shaping in a hierarchical working memory task. It shows that shaping significantly boosts the speed and robustness of learning, and the generalization ability. It also proposes a mechanism for automatic resource allocation and shows that the shaping can be effective without extra external intervention.

Main Contributions

The paper introduces a computational treatment of shaping in the context of a cognitive task.
It shows that shaping significantly boosts the speed of acquisition of the task compared with conventional training.
It leads to internal representations that are more robust to task manipulations such as reversals.
It uses the model to investigate some of the elements of successful shaping.
It highlights the importance of the allocation of resources in shaping, and assess the improvements in training times that come from the external guidance, as a function of parametric task complexity.

Abstract

Humans and animals can perform much more complex tasks than they can acquire using pure trial and error learning. This gap is filled by teaching. One important method of instruction is shaping, in which a teacher decomposes a complete task into sub-components, thereby providing an easier path to learning. Despite its importance, shaping has not been substantially studied in the context of computational modeling of cognitive learning. Here we study the shaping of a hierarchical working memory task using an abstract neural network model as the target learner. Shaping significantly boosts the speed of acquisition of the task compared with conventional training, to a degree that increases with the temporal complexity of the task. Further, it leads to internal representations that are more robust to task manipulations such as reversals. We use the model to investigate some of the elements of successful shaping.

Citation Graph

Loading graph...

References [50]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[3]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[4]Adaptive Mixtures of Local Experts

Robert A. Jacobs, Michael I. Jordan, S. J. Nowlan, Geoffrey E. Hinton - 1991

5 papers in library cite

Very cool idea and it's being used again! It's very early so application is a bit meh, and the part about speech recognition is a bit hard to grasp without context. However, very well explained!

[5]Learning and Development in Neural Networks: The Importance of Starting Small

Jeffrey L. Elman - 1993

5 papers in library cite

This is such a nice paper! Maybe because it's written for a specific public, but it's such an easy read and ties back a lot with neural/biological concepts!

[6]A Day of Great Illumination: B. F. Skinner's Discovery of Shaping

G. B. Peterson - 2004

3 papers in library cite

[7]Constraints on Learning and Their Role in Language Acquisition: Studies of the Acquisition of American Sign Language

E. L. Newport - 1988

2 papers in library cite

[8]How Does a Brain Build a Cognitive Code?

S. Grossberg - 1980

2 papers in library cite

[9]Language Acquisition in the Absence of Explicit Negative Evidence: How Important Is Starting Small?

D. Rohde, D. Plaut - 1999

2 papers in library cite

[10]Learning From Delayed Rewards

C. J. C. H. Watkins - 1989

2 papers in library cite

[11]Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia

R. C. O'reilly, M. J. Frank - 2005

2 papers in library cite

[12]Maturational Constraints on Language Learning

E. L. Newport - 1990

2 papers in library cite

[13]Recent Advances in Hierarchical Reinforcement Learning

A. G. Barto, S. Mahadevan - 2003

2 papers in library cite

[14]A Computational Model of Event Segmentation From Perceptual Prediction

J. R. Reynolds, J. M. Zacks, T. S. Braver - 2007

1 paper in library cites

[15]An Information Theoretical Approach to Prefrontal Executive Function

E. Koechlin, C. Summerfield - 2007

1 paper in library cites

[16]Attention, Intelligence, and the Frontal Lobes

J. Duncan - 1995

1 paper in library cites

[17]Auto-Maintenance in the Pigeon: Sustained Pecking Despite Contingent Non-Reinforcement

D. R. Williams, H. Williams - 1969

1 paper in library cites

[18]Automatic Discovery of Subgoals in Reinforcement Learning Using Diverse Density

A. Mcgovern, A. G. Barto - 2001

1 paper in library cites

[19]Deficits in Strategy Application Following Frontal Lobe Damage in Man

T. Shallice, P. W. Burgess - 1991

1 paper in library cites

[20]Dissociable Controlled Retrieval and Generalized Selection Mechanisms in Ventrolateral Prefrontal Cortex

D. Badre, R. A. Poldrack, E. P. Blagoev, R. Z. J. A. Insler, A. D. Wagner - 2005

1 paper in library cites

[21]Encoding a Temporally Structured Stimulus With a Temporally Structured Neural Representation

S. L. Brown, J. Joseph, M. Stopfer - 2005

1 paper in library cites

[22]Event Perception: A Mind-Brain Perspective

J. M. Zacks, N. K. Speer, K. M. Swallow, T. S. Braver, J. R. Reynolds - 2007

1 paper in library cites

[23]Generative Models for Discovering Sparse Distributed Representations

Geoffrey E. Hinton, Zoubin Ghahramani - 1997

1 paper in library cites

[24]Hierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization

B. Bakker, Jürgen Schmidhuber - 2004

1 paper in library cites

[25]Images, Frames, and Connectionist Hierarchies

Peter Dayan - 2006

1 paper in library cites

[26]In Vivo Evidence for Post-Adolescent Brain Maturation in Frontal and Striatal Regions

E. R. Sowell, P. M. Thompson, C. Holmes, T. L. Jernigan, A. W. Toga - 1999

1 paper in library cites

[27]Interactions Between Frontal Cortex and Basal Ganglia in Working Memory: A Computational Model

M. J. Frank, B. Loughry, R. C. O'reilly - 2001

1 paper in library cites

[28]Involvement of Rostral Prefrontal Cortex in Selection Between Stimulus-Oriented and Stimulus-Independent Thought

S. J. Gilbert, C. D. Frith, P. W. Burgess - 2005

1 paper in library cites

[29]Language-Naive Chimpanzees (Pan troglodytes) judge Relations Between Relations in a Conceptual Matching-to-Sample Task

R. Thompson, D. Oden, S. Boysen - 1997

1 paper in library cites

[30]Limbic Lesions and the Problem of Stimulus–reinforcement Associations

B. Jones, M. Mishkin - 1972

1 paper in library cites

[31]MOSAIC Model for Sensorimotor Learning and Control

M. Haruno, D. M. Wolpert, M. Kawato - 2001

1 paper in library cites

[32]Perseveration in Extinction and in Discrimination Reversal Tasks Following Selective Frontal Ablations in Macaca mulatta

C. M. Butter - 1969

1 paper in library cites

[33]Perseverative Interference in Monkeys Following Selective Lesions of the Inferior Prefrontal Convexity

S. D. Iversen, M. Mishkin - 1970

1 paper in library cites

[34]Phasic Norepinephrine: A Neural Interrupt Signal for Unexpected Events

Peter Dayan, A. J. Yu - 2006

1 paper in library cites

[35]Prefrontal Cortex and Basal ganglia Control Access to Working Memory

F. Mcnab, T. Klingberg - 2008

1 paper in library cites

[36]Reconciling Reinforcement Learning Models With Behavioral Extinction and Renewal: Implications for Addiction, Relapse, and Problem Gambling

A. D. Redish, S. Jensen, A. Johnson, Z. K. Nelson - 2007

1 paper in library cites

[37]Reinforcement Learning With Hierarchies of Machines

R. Parr, S. Russell - 1997

1 paper in library cites

[38]Relation Between Classical Conditioning and Instrumental Learning

F. Sheffield - 1965

1 paper in library cites

[39]Robot Shaping: An Experiment in Behavior Engineering

M. Dorigo, M. Colombetti - 1998

1 paper in library cites

[40]Shaping Robot Behavior Using Principles From Instrumental Conditioning

L. M. Saksida, S. M. Raymond, D. S. Touretzky - 1997

1 paper in library cites

[41]Shaping: A Multiple Contingencies Analysis and Its Relevance to Behaviour-Based Robotics

T. Savage - 2001

1 paper in library cites

[42]Shaping: The Link Between Rats and Robots

T. Savage - 1998

1 paper in library cites

[43]The Architecture of Cognitive Control in the Human Prefrontal Cortex

E. Koechlin, C. Ody, F. Kouneiher - 2003

1 paper in library cites

[44]The Behavior of Organisms: An Experimental Analysis

B. F. Skinner - 1938

1 paper in library cites

[45]The Codes of Man and Beasts

D. Premack - 1983

1 paper in library cites

[46]The Misbehavior of Organisms

K. Breland, M. Breland - 1961

1 paper in library cites

[47]Towards an Executive Without a Homunculus: Computational Models of the Prefrontal Cortex/Basal Ganglia System

T. E. Hazy, M. J. Frank, R. C. O'reilly - 2007

1 paper in library cites

[48]Transfer of Learning After Updating Training Mediated by the Striatum

E. Dahlin, A. S. Neely, A. Larsson, L. Backman, L. Nyberg - 2008

1 paper in library cites

[49]Transfer of Learning by Composing Solutions of Elemental Sequential Tasks

S. P. Singh - 1992

1 paper in library cites

[50]Uncertainty, Neuromodulation, and Attention

A. J. Yu, Peter Dayan - 2005

1 paper in library cites

Cited by

1

papers in your library

Cites

5

papers in your library

Read

on July 12, 2025

Good paper overall, fun to read, but it's just a toy example so results are very meh.

Tags

Paper Aliases

No aliases