Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

Ryan Clancy, Nicola Ferro, Claudia Hauff, Jimmy Lin, Tetsuya Sakai, Ze Zhong Wu

Research output: Contribution to journalConference article

Abstract

The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

Original languageEnglish
Pages (from-to)1-7
Number of pages7
JournalCEUR Workshop Proceedings
Volume2409
Publication statusPublished - 2019 Jan 1
Event2019 Open-Source IR Replicability Challenge, OSIRRC 2019 - Paris, France
Duration: 2019 Jul 25 → …

Fingerprint

Specifications
Jigs
Information retrieval
Experiments

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Clancy, R., Ferro, N., Hauff, C., Lin, J., Sakai, T., & Wu, Z. Z. (2019). Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). CEUR Workshop Proceedings, 2409, 1-7.

Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). / Clancy, Ryan; Ferro, Nicola; Hauff, Claudia; Lin, Jimmy; Sakai, Tetsuya; Wu, Ze Zhong.

In: CEUR Workshop Proceedings, Vol. 2409, 01.01.2019, p. 1-7.

Research output: Contribution to journalConference article

Clancy, R, Ferro, N, Hauff, C, Lin, J, Sakai, T & Wu, ZZ 2019, 'Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)', CEUR Workshop Proceedings, vol. 2409, pp. 1-7.
Clancy, Ryan ; Ferro, Nicola ; Hauff, Claudia ; Lin, Jimmy ; Sakai, Tetsuya ; Wu, Ze Zhong. / Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019). In: CEUR Workshop Proceedings. 2019 ; Vol. 2409. pp. 1-7.
@article{b2d00ff0652c4214b86b4171cd381ed2,
title = "Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)",
abstract = "The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the {"}jig{"} that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.",
author = "Ryan Clancy and Nicola Ferro and Claudia Hauff and Jimmy Lin and Tetsuya Sakai and Wu, {Ze Zhong}",
year = "2019",
month = "1",
day = "1",
language = "English",
volume = "2409",
pages = "1--7",
journal = "CEUR Workshop Proceedings",
issn = "1613-0073",
publisher = "CEUR-WS",

}

TY - JOUR

T1 - Overview of the 2019 open-source IR replicability challenge (OSIRRC 2019)

AU - Clancy, Ryan

AU - Ferro, Nicola

AU - Hauff, Claudia

AU - Lin, Jimmy

AU - Sakai, Tetsuya

AU - Wu, Ze Zhong

PY - 2019/1/1

Y1 - 2019/1/1

N2 - The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

AB - The Open-Source IR Replicability Challenge (OSIRRC 2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by gathering a community of researchers to jointly develop a common Docker specification and build Docker images that encapsulate a diversity of systems and retrieval models. We articulate the goals of this workshop and describe the "jig" that encodes the Docker specification. In total, 13 teams from around the world submitted 17 images, most of which were designed to produce retrieval runs for the TREC 2004 Robust Track test collection. This exercise demonstrates the feasibility of orchestrating large, community-based replication experiments with Docker technology. We envision OSIRRC becoming an ongoing community-wide effort to ensure experimental replicability and sustained progress on standard test collections.

UR - http://www.scopus.com/inward/record.url?scp=85070667049&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070667049&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85070667049

VL - 2409

SP - 1

EP - 7

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

SN - 1613-0073

ER -