Sciweavers

EMNLP
2009

Automatically Evaluating Content Selection in Summarization without Human Models

13 years 1 months ago
Automatically Evaluating Content Selection in Summarization without Human Models
We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, JensenShannon divergence, leads to a correlation as high as 0.88 with manual pyramid and 0.73 with responsiveness evaluations.
Annie Louis, Ani Nenkova
Added 17 Feb 2011
Updated 17 Feb 2011
Type Journal
Year 2009
Where EMNLP
Authors Annie Louis, Ani Nenkova
Comments (0)