Papperoni

1997

Multitask Learning

Rich Caruana

Open PDF Google Scholar

citations

Cite Score

86

AI summary

This paper introduces Multi-Task Learning (MTL) to improve generalization by leveraging training signals from related tasks. The method learns tasks in parallel using a shared representation, demonstrated in backpropagation networks, k-nearest neighbor, and kernel regression, achieving improved performance in multiple domains.

Main Contributions

Introduces a Multi-Task Learning (MTL) approach for inductive transfer.
Demonstrates that MTL discovers task relatedness in backpropagation networks without supervisory signals.
Presents new results for MTL with k-nearest neighbor and kernel regression.
Explains how MTL works and identifies opportunities for its application in real domains.
Sketches an algorithm for multitask learning in decision trees.

Abstract

Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better. This paper reviews prior work on MTL, presents new evidence that MTL in backprop nets discovers task relatedness without the need of supervisory signals, and presents new results for MTL with k-nearest neighbor and kernel regression. In this paper we demonstrate multitask learning in three domains. We explain how multitask learning works, and show that there are many opportunities for multitask learning in real domains. We present an algorithm and results for multitask learning with case-based methods like k-nearest neighbor and kernel regression, and sketch an algorithm for multitask learning in decision trees. Because multitask learning works, can be applied to many different kinds of domains, and can be used with different learning algorithms, we conjecture there will be many opportunities for its use on real-world problems.

Citation Graph

Loading graph...

References [53]

Sort:

Filter:

[1]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[2]Backpropagation Applied to Handwritten Zip-Code Recognition

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackal - 1989

24 papers in library cite

The first convolution NN! Very simple concept and very simply explained. Very good results and overall a good read.

[3]Multitask Learning

Rich Caruana - 1997

13 papers in library cite

I expected waaaaaay more from this paper. The idea is sooooo simple and the results are underwhelming. Also, 30 pages for something that could be said in 10. The writing style is a bit boring. TBH it seems like it's just a re-writing of Caruana's PhD thesis.

[4]Hierarchical Mixtures of Experts and the EM Algorithm

M. Jordan, Rowan Jacobs - 1994

3 papers in library cite

I think the main contribution here is the introduction + using the MoE. The rest of the paper is just a bunch of math proofs.

[5]Learning Distributed Representations of Concepts

Geoffrey E. Hinton - 1986

13 papers in library cite

 Probably seminal, but a bit boring overall

[6]The Need for Biases in Learning Generalizations

Tom M. Mitchell - 1980

2 papers in library cite

Somewhat abstract but the main idea is amazing!

[7]Learning Many Related Tasks at the Same Time With Backpropagation

Rich Caruana - 1995

3 papers in library cite

Very nice read, fast and simple. Very well and intuitive explanation as well. However, it doesn't seem too impactful overall (the other multitask learning paper seems more relevant, but also more boring)

[8]Learning From Hints in Neural Networks

Yaser S. Abu Mostafa - 1990

3 papers in library cite

Very nice read, but the results are very underwhelming. I hoped for some more concrete concepts/results.

[9]Direct Transfer of Learned Information Among Neural Networks

Lorien Y. Pratt, Jack Mostow, Candace A. Kamm - 1991

3 papers in library cite

Boring read and uninteresting results. I think they stumbled onto transfer learning but didn't know it yet, so they explored it in a bad way.

[10]Multitask Connectionist Learning

Rich Caruana - 1994

3 papers in library cite

I couldn't find it anywhere

[11]NETtalk: A Parallel Network That Learns to Read Aloud

T. J. Sejnowski, C. R. Rosenberg - 1986

6 papers in library cite

[12]Tangent Prop-A Formalism for Specifying Selected Invariances in an Adaptive Neural Network

Patrice Simard, B. Victorri, Yann Lecun, John Denker - 1992

4 papers in library cite

[13]Learning Internal Representations

J. Baxter - 1994

3 papers in library cite

[14]Multitask Learning: A Knowledge-Based Source of Inductive Bias

Rich Caruana - 1993

3 papers in library cite

[15]A Comparative Study of ID3 and Backpropagation for English Text-to-Speech Mapping

T. G. Dietterich, H. Hild, G. Bakiri - 1990

2 papers in library cite

[16]A Comparative Study of ID3 and Backpropagation for English Text-to-Speech Mapping

T. G. Dietterich, H. Hild, G. Bakiri - 1990

2 papers in library cite

[17]A Personal Learning Apprentice

L. Dent, J. Boticario, J. Mcdermott, Tom M. Mitchell, D. Zabowski - 1992

2 papers in library cite

[18]Causation, Prediction, and Search

P. Spirtes, C. Glymour, R. Scheines - 1993

2 papers in library cite

[19]Hints and the VC Dimension

Yaser S. Abu Mostafa - 1993

2 papers in library cite

[20]Induction of Decision Trees

J. R. Quinlan - 1986

2 papers in library cite

[21]Neural Network Perception for Mobile Robot Guidance

D. A. Pomerleau - 1992

2 papers in library cite

[22]Predicting Multivariate Responses in Multiple Linear Regression

L. Breiman, J. H. Friedman - 1995

2 papers in library cite

[23]Solving Multiclass Learning Problems via Error-Correcting Output Codes

T. G. Dietterich, G. Bakiri - 1995

2 papers in library cite

[24]Using Additive Noise in Back-Propagation Training

L. Holmstrom, P. Koistinen - 1992

2 papers in library cite

[25]A Bayesian Method for the Induction of Probabilistic Networks From Data

G. F. Cooper, E. Herskovits - 1992

1 paper in library cites

[26]A Bayesian/Information Theoretic Model of Bias Learning

J. Baxter - 1996

1 paper in library cites

[27]A Probibilistic Approach to Feature Selection-a Filter Solution

Haozhe Liu, R. Setiono - 1996

1 paper in library cites

[28]Acquiring and Combining Overlapping Concepts

J. D. Martin, D. O. Billman - 1994

1 paper in library cites

[29]Adaptive Generalisation and the Transfer of Knowledge

N. E. Sharkey, A. J. C. Sharkey - 1992

1 paper in library cites

[30]An Evaluation of Machine Learning Methods for Predicting Pneumonia Mortality

G. F. Cooper, C. F. Aliferis, R. Ambrosino, J. Aronis, B. G. Buchanan, Rich Caruana, M. J. Fine, C. Glymour, G. Gordon, B. H. Hanusa, J. E. Janosky, C. Meek, Tom M. Mitchell, T. Richardson, P. Spirtes - 1997

1 paper in library cites

[31]C4.5: Programs for Machine Learning

J. R. Quinlan - 1992

1 paper in library cites

[32]Competition Among Networks Improves Committee Performance

P. W. Munro, B. Parmanto - 1997

1 paper in library cites

[33]Conceptual Clustering, Learning From Examples, and Inference

D. H. Fisher - 1987

1 paper in library cites

[34]Discovering Structure in Multiple Learning Tasks: The TC Algorithm

J. O'sullivan, Sebastian Thrun - 1996

1 paper in library cites

[35]Experience With a Learning Personal Assistant

Tom M. Mitchell, Rich Caruana, D. Freitag, J. Mcdermott, D. Zabowski - 1994

1 paper in library cites

[36]Family Discovery

S. M. Omohundro - 1996

1 paper in library cites

[37]Goal-Directed Clustering

J. D. Martin - 1994

1 paper in library cites

Yaser S. Abu Mostafa - 1995

1 paper in library cites

[39]Learning Classification With Unlabelled Data

V. R. D. Sa - 1994

1 paper in library cites

[40]Mixture Models for Learning From Incomplete Data

Zoubin Ghahramani, Michael I. Jordan - 1997

1 paper in library cites

[41]Monotonicity Hints

J. Sill, Y. A. Mostafa - 1997

1 paper in library cites

[42]Multi-Task Learning for Stock Selection

J. Ghosn, Yoshua Bengio - 1997

1 paper in library cites

[43]Non-Literal Transfer Among Neural Network Learners

Lorien Y. Pratt - 1992

1 paper in library cites

[44]Promoting Poor Features to Supervisors: Some Inputs Work Better as Outputs

Rich Caruana, V. R. D. Sa - 1997

1 paper in library cites

[45]Rule-Injection Hints as a Means of Improving Network Performance and Learning Time

S. C. Suddarth, Y. L. Kergosien - 1990

1 paper in library cites

[46]Sensor Fusion for Autonomous Outdoor Navigation Using Neural Networks

I. Davis, A. Stentz - 1995

1 paper in library cites

[47]Statistical Analysis With Missing Data

R. J. A. Little, D. B. Rubin - 1987

1 paper in library cites

[48]Supervised Learning From Incomplete Data Using an EM Approach

Zoubin Ghahramani, Michael I. Jordan - 1994

1 paper in library cites

[49]Toward Optimal Feature Selection

D. Koller, M. Sahami - 1996

1 paper in library cites

[50]Using Sampling and Queries to Extract Rules From Trained Neural Networks

M. Craven, J. Shavlik - 1994

1 paper in library cites

[51]Using the Future to "Sort Out" the Present: Rankprop and Multitask Learning for Medical Risk Prediction

Rich Caruana, S. Baluja, Tom M. Mitchell - 1996

1 paper in library cites

[52]Using the Representation in a Neural Network's Hidden Layer for Task-Specific Focus of Attention

S. Baluja, D. A. Pomerleau - 1995

1 paper in library cites

[53]Validation of a Pneumonia Prognostic Index Using the MedisGroups Comparative Hospital Database

M. J. Fine, D. Singer, B. H. Hanusa, J. Lave, W. Kapoor - 1993

1 paper in library cites

Cited by

13

papers in your library

Cites

9

papers in your library

Read

on July 18, 2025

I expected waaaaaay more from this paper. The idea is sooooo simple and the results are underwhelming. Also, 30 pages for something that could be said in 10. The writing style is a bit boring. TBH it seems like it's just a re-writing of Caruana's PhD thesis.

Tags

Paper Aliases

No aliases