Metrics, statistics, tests

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    23 Citations (Scopus)

    Abstract

    This lecture is intended to serve as an introduction to Information Retrieval (IR) effectiveness metrics and their usage in IR experiments using test collections. Evaluation metrics are important because they are inexpensive tools for monitoring technological advances. This lecture covers a wide variety of IR metrics (except for those designed for XML retrieval, as there is a separature lecture dedicated to this topic) and discusses some methods for evaluating evaluation metrics. It also briefly covers computer-based statistical significance testing. The takeaways for IR experimenters are: (1) It is important to understand the properties of IR metrics and choose or design appropriate ones for the task at hand; (2) Computer-based statistical significance tests are simple and useful, although statistical significance does not necessarily imply practical significance, and statistical insignificance does not necessarily imply practical insignificance; and (3) Several methods exist for discussing which metrics are "good," although none of them is perfect.

    Original languageEnglish
    Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    PublisherSpringer Verlag
    Pages116-163
    Number of pages48
    Volume8173 LNCS
    ISBN (Print)9783642547973
    DOIs
    Publication statusPublished - 2014
    Event2013 PROMISE Winter School: Bridging Between Information Retrieval and Databases - Bressanone
    Duration: 2013 Feb 42013 Feb 8

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume8173 LNCS
    ISSN (Print)03029743
    ISSN (Electronic)16113349

    Other

    Other2013 PROMISE Winter School: Bridging Between Information Retrieval and Databases
    CityBressanone
    Period13/2/413/2/8

    Fingerprint

    Information retrieval
    Test Statistic
    Information Retrieval
    Statistics
    Metric
    Statistical Significance
    Cover
    Statistical tests
    Imply
    Significance Test
    XML
    Evaluation
    Statistical test
    Retrieval
    Choose
    Monitoring
    Testing
    Experiments
    Experiment

    ASJC Scopus subject areas

    • Computer Science(all)
    • Theoretical Computer Science

    Cite this

    Sakai, T. (2014). Metrics, statistics, tests. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8173 LNCS, pp. 116-163). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8173 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-642-54798-0_6

    Metrics, statistics, tests. / Sakai, Tetsuya.

    Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8173 LNCS Springer Verlag, 2014. p. 116-163 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8173 LNCS).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Sakai, T 2014, Metrics, statistics, tests. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 8173 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8173 LNCS, Springer Verlag, pp. 116-163, 2013 PROMISE Winter School: Bridging Between Information Retrieval and Databases, Bressanone, 13/2/4. https://doi.org/10.1007/978-3-642-54798-0_6
    Sakai T. Metrics, statistics, tests. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8173 LNCS. Springer Verlag. 2014. p. 116-163. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-642-54798-0_6
    Sakai, Tetsuya. / Metrics, statistics, tests. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 8173 LNCS Springer Verlag, 2014. pp. 116-163 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
    @inproceedings{f5f0318dacc04d6492bdf2d6b79bdb06,
    title = "Metrics, statistics, tests",
    abstract = "This lecture is intended to serve as an introduction to Information Retrieval (IR) effectiveness metrics and their usage in IR experiments using test collections. Evaluation metrics are important because they are inexpensive tools for monitoring technological advances. This lecture covers a wide variety of IR metrics (except for those designed for XML retrieval, as there is a separature lecture dedicated to this topic) and discusses some methods for evaluating evaluation metrics. It also briefly covers computer-based statistical significance testing. The takeaways for IR experimenters are: (1) It is important to understand the properties of IR metrics and choose or design appropriate ones for the task at hand; (2) Computer-based statistical significance tests are simple and useful, although statistical significance does not necessarily imply practical significance, and statistical insignificance does not necessarily imply practical insignificance; and (3) Several methods exist for discussing which metrics are {"}good,{"} although none of them is perfect.",
    author = "Tetsuya Sakai",
    year = "2014",
    doi = "10.1007/978-3-642-54798-0_6",
    language = "English",
    isbn = "9783642547973",
    volume = "8173 LNCS",
    series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
    publisher = "Springer Verlag",
    pages = "116--163",
    booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

    }

    TY - GEN

    T1 - Metrics, statistics, tests

    AU - Sakai, Tetsuya

    PY - 2014

    Y1 - 2014

    N2 - This lecture is intended to serve as an introduction to Information Retrieval (IR) effectiveness metrics and their usage in IR experiments using test collections. Evaluation metrics are important because they are inexpensive tools for monitoring technological advances. This lecture covers a wide variety of IR metrics (except for those designed for XML retrieval, as there is a separature lecture dedicated to this topic) and discusses some methods for evaluating evaluation metrics. It also briefly covers computer-based statistical significance testing. The takeaways for IR experimenters are: (1) It is important to understand the properties of IR metrics and choose or design appropriate ones for the task at hand; (2) Computer-based statistical significance tests are simple and useful, although statistical significance does not necessarily imply practical significance, and statistical insignificance does not necessarily imply practical insignificance; and (3) Several methods exist for discussing which metrics are "good," although none of them is perfect.

    AB - This lecture is intended to serve as an introduction to Information Retrieval (IR) effectiveness metrics and their usage in IR experiments using test collections. Evaluation metrics are important because they are inexpensive tools for monitoring technological advances. This lecture covers a wide variety of IR metrics (except for those designed for XML retrieval, as there is a separature lecture dedicated to this topic) and discusses some methods for evaluating evaluation metrics. It also briefly covers computer-based statistical significance testing. The takeaways for IR experimenters are: (1) It is important to understand the properties of IR metrics and choose or design appropriate ones for the task at hand; (2) Computer-based statistical significance tests are simple and useful, although statistical significance does not necessarily imply practical significance, and statistical insignificance does not necessarily imply practical insignificance; and (3) Several methods exist for discussing which metrics are "good," although none of them is perfect.

    UR - http://www.scopus.com/inward/record.url?scp=84901306933&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84901306933&partnerID=8YFLogxK

    U2 - 10.1007/978-3-642-54798-0_6

    DO - 10.1007/978-3-642-54798-0_6

    M3 - Conference contribution

    AN - SCOPUS:84901306933

    SN - 9783642547973

    VL - 8173 LNCS

    T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    SP - 116

    EP - 163

    BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    PB - Springer Verlag

    ER -