Papperoni

1994

Hierarchical Mixtures of Experts and the EM Algorithm

M. Jordan, Rowan Jacobs

citations

Cite Score

AI summary

This paper introduces a tree-structured architecture for supervised learning, utilizing hierarchical mixture models and the Expectation-Maximization (EM) algorithm for parameter adjustment and on-line learning, demonstrating its effectiveness in robot dynamics.

Main Contributions

Proposed a tree-structured architecture called Hierarchical Mixture-of-Experts (HME) for supervised learning.
Formulated the learning problem as a maximum likelihood problem within a hierarchical mixture model framework.
Derived an Expectation-Maximization (EM) algorithm for adjusting the parameters of the HME architecture, treating it as iterative-reweighted least-squares problems.
Developed an on-line learning algorithm for HME using recursive estimation theory.
Showcased comparative simulation results in robot dynamics, demonstrating faster convergence compared to backpropagation and better accuracy than CART and MARS.

Abstract

We present a tree-structured architecture for supervised learning. The statistical model underlying the architecture is a hierarchical mixture model in which both the mixture coefficients and the mixture components are generalized linear models (GLIM's). Learning is treated as a maximum likelihood problem; in particular, we present an Expectation-Maximization (EM) algorithm for adjusting the parameters of the architecture. We also develop an on-line learning algorithm in which the parameters are updated incrementally. Comparative simulation results are presented in the robot dynamics domain.

Citation Graph

Loading graph...

References [12]

Sort:

Filter:

[1]Adaptive Mixtures of Local Experts

Robert A. Jacobs, Michael I. Jordan, S. J. Nowlan, Geoffrey E. Hinton - 1991

5 papers in library cite

Google Scholar

Very cool idea and it's being used again! It's very early so application is a bit meh, and the part about speech recognition is a bit hard to grasp without context. However, very well explained!

[2]Hierarchical Mixtures of Experts and the EM Algorithm

M. Jordan, Rowan Jacobs - 1994

3 papers in library cite

Google Scholar

I think the main contribution here is the introduction + using the MoE. The rest of the paper is just a bunch of math proofs.

[3]Probabilistic Interpretation of Feedforward Classification Network Outputs, With Relationship to Statistical Pattern Recognition

John S. Bridle - 1990

12 papers in library cite

Google Scholar

Explains using softmax as output. Nice read, but I am not sure if too relevant.

[4]Maximum Likelihood From Incomplete Data via the EM Algorithm

A. P. Dempster, N. M. Laird, D. B. Rubin - 1977

6 papers in library cite

Google Scholar

[5]Classification and Regression Trees

L. Breiman, J. H. Friedman, R. A. Olshen, C. J. Stone - 1984

2 papers in library cite

Google Scholar

[6]Induction of Decision Trees

J. R. Quinlan - 1986

2 papers in library cite

Google Scholar

[7]Autoclass: A Bayesian Classification System

P. Cheeseman, J. Kelly, M. Self, J. Stutz, W. Taylor, Daniel Freeman - 1988

1 paper in library cites

Google Scholar

[8]Convergence Results for the EM Approach to Mixtures-of-Experts Architectures

Michael I. Jordan, L. Xu - 1993

1 paper in library cites

Google Scholar

[9]Generalized Linear Models

P. Mccullagh, J. A. Nelder - 1983

1 paper in library cites

Google Scholar

[10]Hierarchies of Adaptive Experts

Michael I. Jordan, Robert A. Jacobs - 1992

1 paper in library cites

Google Scholar

[11]Multivariate Adaptive Regression Splines

J. H. Friedman - 1991

1 paper in library cites

Google Scholar

[12]Theory and Practice of Recursive Identification

L. Ljung, T. Soderstrom - 1986

1 paper in library cites

Google Scholar

Cited by

papers in your library

Cites

papers in your library

Read

on January 19, 2026

I think the main contribution here is the introduction + using the MoE. The rest of the paper is just a bunch of math proofs.