Sciweavers

Share
COLING
2008

Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality

9 years 2 months ago
Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality
In this paper, we analyze the state of current human and automatic evaluation of topic-focused summarization in the Document Understanding Conference main task for 2005-2007. The analyses show that while ROUGE has very strong correlation with responsiveness for both human and automatic summaries, there is a significant gap in responsiveness between humans and systems which is not accounted for by the ROUGE metrics. In addition to teasing out gaps in the current automatic evaluation, we propose a method to maximize the strength of current automatic evaluations by using the method of canonical correlation. We apply this new evaluation method, which we call ROSE (ROUGE Optimal Summarization Evaluation), to find the optimal linear combination of ROUGE scores to maximize correlation with human responsiveness.
John M. Conroy, Hoa Trang Dang
Added 29 Oct 2010
Updated 29 Oct 2010
Type Conference
Year 2008
Where COLING
Authors John M. Conroy, Hoa Trang Dang
Comments (0)
books