Toward Qualitative Evaluation of Textual Entailment Systems

This paper presents a methodology for a quantitative and qualitative evaluation of Textual Entailment systems. We take advantage of the decomposition of Text Hypothesis pairs into monothematic pairs, i.e. pairs where only one linguistic phenomenon at a time is responsible for entailment judgment, and propose to run TE systems over such datasets. We show that several behaviours of a system can be explained in terms of the correlation between the accuracy on monothematic pairs and the accuracy on the corresponding original pairs.

• Type: Journal
• Year: 2010
• Where: COLING
• Authors: Elena Cabrio, Bernardo Magnini
• Keywords: COLING 2010, Computational Linguistics, Corresponding Original Pairs, Monothematic Pairs, Textual Entailment Systems