Papperoni

1996

A Maximum Entropy Approach to Natural Language Processing

A. L. Berger, S. A. D. Pietra, Vincent J. Della Pietra

Open PDF Google Scholar

citations

Cite Score

73

AI summary

This paper introduces a maximum-likelihood approach for constructing maximum entropy models for NLP tasks, using improved iterative scaling for parameter estimation and feature selection to maximize the likelihood of training data, and achieving strong results on bilingual sense disambiguation, word reordering, and sentence segmentation.

Main Contributions

Presents a maximum-likelihood approach for automatically constructing maximum entropy models.
Describes how to implement this approach efficiently using examples in natural language processing.
Introduces an automatic method for feature selection to be included in a maximum entropy model.
Applies maximum entropy ideas to several tasks in stochastic language processing such as bilingual sense disambiguation, word reordering, and sentence segmentation.
Demonstrates the efficacy of maximum entropy techniques for performing context-sensitive modeling.

Abstract

The concept of maximum entropy can be traced back along multiple threads to Biblical times. Only recently, however, have computers become powerful enough to permit the widescale application of this concept to real world problems in statistical estimation and pattern recognition. In this paper, we describe a method for statistical modeling based on maximum entropy. We present a maximum-likelihood approach for automatically constructing maximum entropy models and describe how to implement this approach efficiently, using as examples several problems in natural language processing.

Citation Graph

Loading graph...

References [23]

Sort:

Filter:

[1]Class-Based N-Gram Models of Natural Language

P. F. Brown, P. V. Desouza, R. L. Mercer, Vincent J. Della Pietra, J. C. Lai - 1992

12 papers in library cite

[2]Interpolated Estimation of Markov Source Parameters From Sparse Data

Frederick Jelinek, R. L. Mercer - 1980

8 papers in library cite

[3]The Mathematics of Statistical Machine Translation: Parameter Estimation

P. F. Brown, S. D. Pietra, Vincent J. Della Pietra, R. L. Mercer - 1993

7 papers in library cite

[4]Maximum Likelihood From Incomplete Data via the EM Algorithm

A. P. Dempster, N. M. Laird, D. B. Rubin - 1977

6 papers in library cite

[5]A Statistical Approach to Machine Translation

P. Brown, J. Cocke, S. D. Pietra, Vincent J. Della Pietra, Frederick Jelinek, J. Lafferty, R. Mercer, P. Roossin - 1990

3 papers in library cite

[6]Elements of Information Theory

T. Cover, J. Thomas - 1991

2 papers in library cite

[7]A Geometric Interpretation of Darroch and Ratcliff's Generalized Iterative Scaling

I. Csiszar - 1989

1 paper in library cites

[8]A Note on Approximations to Discrete Probability Distributions

D. Brown - 1959

1 paper in library cites

[9]A Statistical Approach to Sense Disambiguation in Machine Translation

P. Brown, S. D. Pietra, V. D. Pietra, R. Mercer - 1991

1 paper in library cites

[10]A Tree-Based Statistical Language Model for Natural Language Speech Recognition

L. Bahl, P. Brown, P. D. Souza, R. Mercer - 1989

1 paper in library cites

[11]An Information Theoretic Approach to Automatic Determination of Phonemic Baseforms

J. Lucassen, R. Mercer - 1984

1 paper in library cites

[12]Continuous Speech Recognition With Automatically Selected Acoustic Prototypes Obtained by Either Bootstrapping or Clustering

A. Nadas, R. Mercer, L. Bahl, R. Bakis, Paul Cohen, A. Cole, Frederick Jelinek, B. Lewis - 1981

1 paper in library cites

[13]Generalized Iterative Scaling for Log-Linear Models

J. N. Darroch, D. Ratcliff - 1972

1 paper in library cites

[14]I-Divergence Geometry of Probability Distributions and Minimization Problems

I. Csiszar - 1975

1 paper in library cites

[15]Inducing Features of Random Fields"

S. D. Pietra, V. D. Pietra, J. Lafferty - 1995

1 paper in library cites

[16]Inference and Estimation of a Long-Range Trigram Model

S. D. Pietra, V. D. Pietra, J. Gillett, J. Lafferty, H. Printz, L. Ures - 1994

1 paper in library cites

[17]Information Geometry and Alternating Minimization Procedures

L. Csiszar, G. Tusnady - 1984

1 paper in library cites

[18]Mathematics of Physics and Modern Engineering

I. S. Sokolnikoff, R. M. Redheffer - 1966

1 paper in library cites

[19]Notes on Present Status and Future Prospects

E. T. Jaynes - 1990

1 paper in library cites

[20]Tagging Text With a Probabilistic Model

B. Merialdo - 1990

1 paper in library cites

[21]The Candide System for Machine Translation

A. Berger, P. Brown, S. D. Pietra, V. D. Pietra, J. Gillett, J. Lafferty, H. Printz, L. Ures - 1994

1 paper in library cites

[22]The Principle of Maximum Entropy

S. Guiasu, A. Shenitzer - 1985

1 paper in library cites

[23]Towards History-Based Grammars: Using Richer Models for Probabilistic Parsing

E. Black, Frederick Jelinek, J. Lafferty, D. Magerman, R. Mercer, S. Roukos - 1992

1 paper in library cites

Cited by

10

papers in your library

Cites

0

papers in your library

Read

on June 30, 2025

This paper is so good! Easy to follow and very nice results. The experiments are a bit meh, but otherwise wonderful.

Tags

Paper Aliases

No aliases