Can we Evaluate the Quality of Generated Text?

Evaluating the output of NLG systems is notoriously difficult, and performing assessments of text quality even more so. A range of automated and subject-based approaches to the evaluation of text quality have been taken, including comparison with a putative gold standard text, analysis of specific linguistic features of the output, expert review and task-based evaluation. In this paper we present the results of a variety of such approaches in the context of a case study application. We discuss the problems encountered in the implementation of each approach in the context of the literature, and propose that a test based on the Turing test for machine intelligence offers a way forward in the evaluation of the subjective notion of text quality.

• Type: Conference
• Year: 2008
• Where: LREC
• Authors: David Hardcastle, Donia Scott
• Keywords: Education, Gold Standard Text, LREC 2008, Specific Linguistic Features, Text Quality