22 search results - page 1 / 5 » Assessing the Effect of Inconsistent Assessors on Summarizat... |
ACL
2012
11 years 6 months ago
2012
We investigate the consistency of human assessors involved in summarization evaluation to understand its effect on system ranking and automatic evaluation techniques. Using Text A...
SIGIR
2012
ACM
11 years 6 months ago
2012
ACM
Assessors frequently disagree on the topical relevance of documents. How much of this disagreement is due to ambiguity in assessment instructions? We have two assessors assess TRE...
CIVR
2008
Springer
13 years 6 months ago
2008
Springer
For the first time in 2007, TRECVID considered structured evaluation of automated video summarization, utilizing BBC rushes video. In 2007, we conducted user evaluations with the ...
SIGIR
2005
ACM
13 years 10 months ago
2005
ACM
The effectiveness of information retrieval systems is measured by comparing performance on a common set of queries and documents. Significance tests are often used to evaluate the...
TRECVID
2007
13 years 5 months ago
2007
For the first time in 2007, TRECVID considered structured evaluation of automated video summarization, utilizing BBC rushes video. This paper discusses in detail our approaches fo...