Sciweavers

Share
LRE
2016

SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sen

3 years 14 days ago
SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sen
Abstract This paper is an extended description of SemEval-2014 Task 1, the task on the evaluation of Compositional Distributional Semantics Models on full sentences. Systems participating in the task were presented with pairs of sentences and were evaluated on their ability to predict human judgments on (i) semantic relatedness and (ii) entailment. Training and testing data were subsets of the SICK (Sentences Involving Compositional Knowledge) data set. SICK was developed with the aim of providing a proper benchmark to evaluate compositional semantic systems, though task participation was open to systems based on any approach. Taking advantage of the SemEval experience, in this paper we analyze the SICK data set, in order to evaluate the extent to which it meets its design goal and to shed light on the linguistic phenomena that are still challenging for state-of-the-art computational semantic systems. Qualitative and quantitative error analyses show that many systems are quite sensitiv...
Luisa Bentivogli, Raffaella Bernardi, Marco Marell
Added 07 Apr 2016
Updated 07 Apr 2016
Type Journal
Year 2016
Where LRE
Authors Luisa Bentivogli, Raffaella Bernardi, Marco Marelli, Stefano Menini, Marco Baroni, Roberto Zamparelli
Comments (0)
books