Sciweavers

Share
ECIR
2016
Springer

Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge

4 years 3 months ago
Toward Reproducible Baselines: The Open-Source IR Reproducibility Challenge
The Open-Source IR Reproducibility Challenge brought together developers of open-source search engines to provide reproducible baselines of their systems in a common environment on Amazon EC2. The product is a repository that contains all code necessary to generate competitive ad hoc retrieval baselines, such that with a single script, anyone with a copy of the collection can reproduce the submitted runs. Our vision is that these results would serve as widely accessible points of comparison in future IR research. This project represents an ongoing effort, but we describe the first phase of the challenge that was organized as part of a workshop at SIGIR 2015. We have succeeded modestly so far, achieving our main goals on the Gov2 collection with seven opensource search engines. In this paper, we describe our methodology, share experimental results, and discuss lessons learned as well as next steps.
Jimmy Lin, Matt Crane, Andrew Trotman, Jamie Calla
Added 02 Apr 2016
Updated 02 Apr 2016
Type Journal
Year 2016
Where ECIR
Authors Jimmy Lin, Matt Crane, Andrew Trotman, Jamie Callan, Ishan Chattopadhyaya, John Foley, Grant Ingersoll, Craig MacDonald, Sebastiano Vigna
Comments (0)
books