Spam filter evaluation with imprecise ground truth

11 years 18 days ago
Spam filter evaluation with imprecise ground truth
When trained and evaluated on accurately labeled datasets, online email spam filters are remarkably effective, achieving error rates an order of magnitude better than classifiers in similar applications. But labels acquired from user feedback or third-party adjudication exhibit higher error rates than the best filters – even filters trained using the same source of labels. It is appropriate to use naturally occuring labels – including errors – as training data in evaluating spam filters. Erroneous labels are problematic, however, when used as ground truth to measure filter effectiveness. Any measurement of the filter’s error rate will be augmented and perhaps masked by the label error rate. Using two natural sources of labels, we demonstrate automatic and semi-automatic methods that reduce the influence of labeling errors on evaluation, yielding substantially more precise measurements of true filter error rates. Categories and Subject Descriptors: H.3.3 [Information...
Gordon V. Cormack, Aleksander Kolcz
Added 28 May 2010
Updated 28 May 2010
Type Conference
Year 2009
Authors Gordon V. Cormack, Aleksander Kolcz
Comments (0)