Sciweavers

ACL
2011

Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation

12 years 7 months ago
Neutralizing Linguistically Problematic Annotations in Unsupervised Dependency Parsing Evaluation
Dependency parsing is a central NLP task. In this paper we show that the common evaluation for unsupervised dependency parsing is highly sensitive to problematic annotations. We show that for three leading unsupervised parsers (Klein and Manning, 2004; Cohen and Smith, 2009; Spitkovsky et al., 2010a), a small set of parameters can be found whose modification yields a significant improvement in standard evaluation measures. These parameters correspond to local cases where no linguistic consensus exists as to the proper gold annotation. Therefore, the standard evaluation does not provide a true indication of algorithm quality. We present a new measure, Neutral Edge Direction (NED), and show that it greatly reduces this undesired phenomenon.
Roy Schwartz, Omri Abend, Roi Reichart, Ari Rappop
Added 23 Aug 2011
Updated 23 Aug 2011
Type Journal
Year 2011
Where ACL
Authors Roy Schwartz, Omri Abend, Roi Reichart, Ari Rappoport
Comments (0)