Topic set size design and power analysis in practice

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    5 Citations (Scopus)

    Abstract

    Topic set size design methods provide principles and procedures for test collection builders to decide on the number of topics to create. These methods can then help us keep improving the test collection design based on accumulated data. Simple Excel tools are available for such purposes. Post-hoc power analysis tools, available as simple R scripts, can help IR researchers examine the achieved power of a reported experiment and determine future sample sizes for ensuring high power. Thus, for example, underpowered user experiments can be detected, and a larger sample size can be proposed. If used appropriately, these Excel and R tools should be able to provide the IR community with better experimentation practices. The main objective of this tutorial is to let IR researchers familiarise themselves with these tools and understand the basic ideas behind them.

    Original languageEnglish
    Title of host publicationICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval
    PublisherAssociation for Computing Machinery, Inc
    Pages9-10
    Number of pages2
    ISBN (Electronic)9781450344975
    DOIs
    Publication statusPublished - 2016 Sep 12
    Event2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016 - Newark, United States
    Duration: 2016 Sep 122016 Sep 16

    Other

    Other2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016
    CountryUnited States
    CityNewark
    Period16/9/1216/9/16

    Keywords

    • Effect sizes
    • Experimental design
    • Statistical power
    • Statistical significance
    • Test collections
    • Variances

    ASJC Scopus subject areas

    • Information Systems
    • Computer Science (miscellaneous)

    Fingerprint Dive into the research topics of 'Topic set size design and power analysis in practice'. Together they form a unique fingerprint.

  • Cite this

    Sakai, T. (2016). Topic set size design and power analysis in practice. In ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval (pp. 9-10). Association for Computing Machinery, Inc. https://doi.org/10.1145/2970398.2970443