Correlating Human and Automatic Evaluation of a German Surface Realiser

We examine correlations between native speaker judgements on automatically generated German text against automatic evaluation metrics. We look at a number of metrics from the MT and Summarisation communities and find that for a relative ranking task, most automatic metrics perform equally well and have fairly strong correlations to the human judgements. In contrast, on a naturalness judgement task, the General Text Matcher (GTM) tool correlates best overall, although in general, correlation between the human judgements and the automatic metrics was quite weak.

• Type: Journal
• Year: 2009
• Where: ACL
• Authors: Aoife Cahill
• Keywords: ACL 2009, Automatic Evaluation Metrics, Automatic Metrics, Computational Linguistics, Human Judgements