Papperoni

1995

Learning Many Related Tasks at the Same Time With Backpropagation

Rich Caruana

citations

Cite Score

AI summary

This paper introduces multitask backpropagation (MTB) to improve generalization in artificial neural nets by enabling simultaneous learning of related tasks, leveraging five identified mechanisms (data amplification, attribute selection, eavesdropping, representation bias), and demonstrating 20-40% improved generalization on road-following and object-recognition domains.

Main Contributions

Identifies five mechanisms (data amplification, attribute selection, eavesdropping, representation bias) by which multitask backprop improves generalization.
Demonstrates that multitask backprop improves generalization without restricting net capacity or reducing VC-dimension, by combining error gradients for related tasks.
Empirically shows that multitask backprop improves generalization by 20-40% on a simulated road-following domain (1D-ALVINN) and a real object-recognition domain (1D-DOORS).
Suggests that MTB typically generalizes better and learns faster than single task backprop (STB), especially for hard tasks.
Highlights that MTB can be used with naturally related tasks or by creating additional tasks to improve performance on main tasks.

Abstract

Hinton [6] proposed that generalization in artificial neural nets should improve if nets learn to represent the domain's underlying regularities. Abu-Mustafa's hints work [1] shows that the outputs of a backprop net can be used as inputs through which domain-specific information can be given to the net. We extend these ideas by showing that a backprop net learning many related tasks at the same time can use these tasks as inductive bias for each other and thus learn better. We identify five mechanisms by which multitask backprop improves generalization and give empirical evidence that multitask backprop generalizes better in real domains.

Citation Graph

Loading graph...

References [10]

Sort:

Filter:

[1]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

Google Scholar

 Probably seminal, but a bit boring overall

[2]Learning From Hints in Neural Networks

Yaser S. Abu Mostafa - 1990

3 papers in library cite

Google Scholar

Very nice read, but the results are very underwhelming. I hoped for some more concrete concepts/results.

[3]Direct Transfer of Learned Information Among Neural Networks

Lorien Y. Pratt, Jack Mostow, Candace A. Kamm - 1991

3 papers in library cite

Google Scholar

Boring read and uninteresting results. I think they stumbled onto transfer learning but didn't know it yet, so they explored it in a bad way.

[4]Multitask Connectionist Learning

Rich Caruana - 1994

3 papers in library cite

Google Scholar

I couldn't find it anywhere

[5]NETtalk: A Parallel Network That Learns to Read Aloud

T. J. Sejnowski, C. R. Rosenberg - 1986

6 papers in library cite

Google Scholar

[6]Symbolic-Neural Systems and the Use of Hints for Developing Complex Systems

S. C. Suddarth, A. D. C. Holden - 1991

3 papers in library cite

Google Scholar

[7]A Comparative Study of ID3 and Backpropagation for English Text-to-Speech Mapping

T. G. Dietterich, H. Hild, G. Bakiri - 1990

2 papers in library cite

Google Scholar

[8]A Personal Learning Apprentice

L. Dent, J. Boticario, J. Mcdermott, Tom M. Mitchell, D. Zabowski - 1992

2 papers in library cite

Google Scholar

[9]Hints and the VC Dimension

Yaser S. Abu Mostafa - 1993

2 papers in library cite

Google Scholar

[10]Neural Network Perception for Mobile Robot Guidance

D. A. Pomerleau - 1992

2 papers in library cite

Google Scholar

Cited by

papers in your library

Cites

papers in your library

Read

on January 22, 2026

Very nice read, fast and simple. Very well and intuitive explanation as well. However, it doesn't seem too impactful overall (the other multitask learning paper seems more relevant, but also more boring)