Sciweavers

Share
ACL
2010

Automatic Evaluation of Linguistic Quality in Multi-Document Summarization

10 years 11 months ago
Automatic Evaluation of Linguistic Quality in Multi-Document Summarization
To date, few attempts have been made to develop and validate methods for automatic evaluation of linguistic quality in text summarization. We present the first systematic assessment of several diverse classes of metrics designed to capture various aspects of well-written text. We train and test linguistic quality models on consecutive years of NIST evaluation data in order to show the generality of results. For grammaticality, the best results come from a set of syntactic features. Focus, coherence and referential clarity are best evaluated by a class of features measuring local coherence on the basis of cosine similarity between sentences, coreference information, and summarization specific features. Our best results are 90% accuracy for pairwise comparisons of competing systems over a test set of several inputs and 70% for ranking summaries of a specific input.
Emily Pitler, Annie Louis, Ani Nenkova
Added 10 Feb 2011
Updated 10 Feb 2011
Type Journal
Year 2010
Where ACL
Authors Emily Pitler, Annie Louis, Ani Nenkova
Comments (0)
books