Papperoni

2004

ROUGE: A Package for Automatic Evaluation of Summaries

Chin Yew Lin

citations

Cite Score

AI summary

This paper introduces ROUGE, a package for automatic evaluation of summaries, which includes four measures: ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S; these measures were evaluated on the DUC 2004 dataset and showed correlation with human judgement.

Main Contributions

Introduces ROUGE, a package for automatic evaluation of summaries.
ROUGE includes four measures: ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S.
ROUGE measures the similarity between summaries by counting overlapping units such as n-grams, word sequences and word pairs.
Evaluates ROUGE measures using DUC 2001, 2002, and 2003 data, showing correlation with human judgments.
Demonstrates the effectiveness of ROUGE package in automatic evaluation of summaries.

Abstract

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper introduces four different ROUGE measures: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S included in the ROUGE summarization evaluation package and their evaluations. Three of them have been used in the Document Understanding Conference (DUC) 2004, a large-scale summarization evaluation sponsored by NIST.

Citation Graph

Loading graph...

References [13]

Sort:

Filter:

[1]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

Google Scholar

Very cool idea. Simple yet very impactful!

[2]Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics

C. Lin, F. Och - 2004

2 papers in library cite

Google Scholar

[3]Automatic Evaluation of Summaries Using N-Gram Co-Occurrence Statistics

Chin Yew Lin, Eduard Hovy - 2003

2 papers in library cite

Google Scholar

[4]Information Retrieval

C. J. V. Rijsbergen - 1979

2 papers in library cite

Google Scholar

[5]Introduction to Algorithms

T. Cormen, C. Leiserson, R. Rivest - 1989

2 papers in library cite

Google Scholar

[6]An Introduction to DUC 2003 - Intrinsic Evaluation of Generic News Text Summarization Systems

P. Over, J. Yen - 2003

1 paper in library cites

Google Scholar

[7]Automatic Evaluation and Uniform Filter Cascades for Inducing N-Best Translation Lexicons

I. Melamed - 1995

1 paper in library cites

Google Scholar

[8]Automatic Summarization

I. Mani - 2001

1 paper in library cites

Google Scholar

[9]Bootstrap Methods and Their Application

A. Davison, D. Hinkley - 1997

1 paper in library cites

Google Scholar

[10]Evaluation of Text Summarization in a Cross-Lingual Information Retrieval Framework

D. R. Radev, S. Teufel, H. Saggion, W. Lam, John Blitzer, A. Gelebi, H. Qi, E. Drabek, D. Liu - 2002

1 paper in library cites

Google Scholar

[11]Looking for a Few Good Metrics: ROUGE and Its Evaluation

C. Lin - 2004

1 paper in library cites

Google Scholar

[12]Meta-Evaluation of Summaries in a Cross-Lingual Environment Using Content-Based Metrics

H. Saggion, D. R. Radev, S. Teufel, W. Lam - 2002

1 paper in library cites

Google Scholar

[13]Precision and Recall of Machine Ranslation

I. Melamed, R. Green, J. Turian - 2003

1 paper in library cites

Google Scholar

Cited by

papers in your library

Cites

papers in your library

Read

on August 7, 2025

Wow, very poorly written and badly explained. This is the exact opposite of BLEU