Metrics, statistics, tests

Research output: Chapter in Book/Report/Conference proceedingConference contribution

25 Citations (Scopus)

Abstract

This lecture is intended to serve as an introduction to Information Retrieval (IR) effectiveness metrics and their usage in IR experiments using test collections. Evaluation metrics are important because they are inexpensive tools for monitoring technological advances. This lecture covers a wide variety of IR metrics (except for those designed for XML retrieval, as there is a separature lecture dedicated to this topic) and discusses some methods for evaluating evaluation metrics. It also briefly covers computer-based statistical significance testing. The takeaways for IR experimenters are: (1) It is important to understand the properties of IR metrics and choose or design appropriate ones for the task at hand; (2) Computer-based statistical significance tests are simple and useful, although statistical significance does not necessarily imply practical significance, and statistical insignificance does not necessarily imply practical insignificance; and (3) Several methods exist for discussing which metrics are "good," although none of them is perfect.

Original languageEnglish
Title of host publicationBridging Between Information Retrieval and Databases - PROMISE Winter School 2013, Revised Tutorial Lectures
PublisherSpringer Verlag
Pages116-163
Number of pages48
ISBN (Print)9783642547973
DOIs
Publication statusPublished - 2014 Jan 1
Event2013 PROMISE Winter School: Bridging Between Information Retrieval and Databases - Bressanone, Italy
Duration: 2013 Feb 42013 Feb 8

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume8173 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference2013 PROMISE Winter School: Bridging Between Information Retrieval and Databases
CountryItaly
CityBressanone
Period13/2/413/2/8

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Metrics, statistics, tests'. Together they form a unique fingerprint.

  • Cite this

    Sakai, T. (2014). Metrics, statistics, tests. In Bridging Between Information Retrieval and Databases - PROMISE Winter School 2013, Revised Tutorial Lectures (pp. 116-163). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8173 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-642-54798-0_6