Topic set size design and power analysis in practice

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    5 Citations (Scopus)

    Abstract

    Topic set size design methods provide principles and procedures for test collection builders to decide on the number of topics to create. These methods can then help us keep improving the test collection design based on accumulated data. Simple Excel tools are available for such purposes. Post-hoc power analysis tools, available as simple R scripts, can help IR researchers examine the achieved power of a reported experiment and determine future sample sizes for ensuring high power. Thus, for example, underpowered user experiments can be detected, and a larger sample size can be proposed. If used appropriately, these Excel and R tools should be able to provide the IR community with better experimentation practices. The main objective of this tutorial is to let IR researchers familiarise themselves with these tools and understand the basic ideas behind them.

    Original languageEnglish
    Title of host publicationICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval
    PublisherAssociation for Computing Machinery, Inc
    Pages9-10
    Number of pages2
    ISBN (Electronic)9781450344975
    DOIs
    Publication statusPublished - 2016 Sep 12
    Event2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016 - Newark, United States
    Duration: 2016 Sep 122016 Sep 16

    Other

    Other2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016
    CountryUnited States
    CityNewark
    Period16/9/1216/9/16

    Fingerprint

    Experiments

    Keywords

    • Effect sizes
    • Experimental design
    • Statistical power
    • Statistical significance
    • Test collections
    • Variances

    ASJC Scopus subject areas

    • Information Systems
    • Computer Science (miscellaneous)

    Cite this

    Sakai, T. (2016). Topic set size design and power analysis in practice. In ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval (pp. 9-10). Association for Computing Machinery, Inc. https://doi.org/10.1145/2970398.2970443

    Topic set size design and power analysis in practice. / Sakai, Tetsuya.

    ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc, 2016. p. 9-10.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Sakai, T 2016, Topic set size design and power analysis in practice. in ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc, pp. 9-10, 2016 ACM International Conference on the Theory of Information Retrieval, ICTIR 2016, Newark, United States, 16/9/12. https://doi.org/10.1145/2970398.2970443
    Sakai T. Topic set size design and power analysis in practice. In ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc. 2016. p. 9-10 https://doi.org/10.1145/2970398.2970443
    Sakai, Tetsuya. / Topic set size design and power analysis in practice. ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc, 2016. pp. 9-10
    @inproceedings{5f362284aa5b4f36b195de94a0a38b9c,
    title = "Topic set size design and power analysis in practice",
    abstract = "Topic set size design methods provide principles and procedures for test collection builders to decide on the number of topics to create. These methods can then help us keep improving the test collection design based on accumulated data. Simple Excel tools are available for such purposes. Post-hoc power analysis tools, available as simple R scripts, can help IR researchers examine the achieved power of a reported experiment and determine future sample sizes for ensuring high power. Thus, for example, underpowered user experiments can be detected, and a larger sample size can be proposed. If used appropriately, these Excel and R tools should be able to provide the IR community with better experimentation practices. The main objective of this tutorial is to let IR researchers familiarise themselves with these tools and understand the basic ideas behind them.",
    keywords = "Effect sizes, Experimental design, Statistical power, Statistical significance, Test collections, Variances",
    author = "Tetsuya Sakai",
    year = "2016",
    month = "9",
    day = "12",
    doi = "10.1145/2970398.2970443",
    language = "English",
    pages = "9--10",
    booktitle = "ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval",
    publisher = "Association for Computing Machinery, Inc",

    }

    TY - GEN

    T1 - Topic set size design and power analysis in practice

    AU - Sakai, Tetsuya

    PY - 2016/9/12

    Y1 - 2016/9/12

    N2 - Topic set size design methods provide principles and procedures for test collection builders to decide on the number of topics to create. These methods can then help us keep improving the test collection design based on accumulated data. Simple Excel tools are available for such purposes. Post-hoc power analysis tools, available as simple R scripts, can help IR researchers examine the achieved power of a reported experiment and determine future sample sizes for ensuring high power. Thus, for example, underpowered user experiments can be detected, and a larger sample size can be proposed. If used appropriately, these Excel and R tools should be able to provide the IR community with better experimentation practices. The main objective of this tutorial is to let IR researchers familiarise themselves with these tools and understand the basic ideas behind them.

    AB - Topic set size design methods provide principles and procedures for test collection builders to decide on the number of topics to create. These methods can then help us keep improving the test collection design based on accumulated data. Simple Excel tools are available for such purposes. Post-hoc power analysis tools, available as simple R scripts, can help IR researchers examine the achieved power of a reported experiment and determine future sample sizes for ensuring high power. Thus, for example, underpowered user experiments can be detected, and a larger sample size can be proposed. If used appropriately, these Excel and R tools should be able to provide the IR community with better experimentation practices. The main objective of this tutorial is to let IR researchers familiarise themselves with these tools and understand the basic ideas behind them.

    KW - Effect sizes

    KW - Experimental design

    KW - Statistical power

    KW - Statistical significance

    KW - Test collections

    KW - Variances

    UR - http://www.scopus.com/inward/record.url?scp=84991098158&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84991098158&partnerID=8YFLogxK

    U2 - 10.1145/2970398.2970443

    DO - 10.1145/2970398.2970443

    M3 - Conference contribution

    AN - SCOPUS:84991098158

    SP - 9

    EP - 10

    BT - ICTIR 2016 - Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval

    PB - Association for Computing Machinery, Inc

    ER -