The effect of score standardisation on topic set size design

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    2 Citations (Scopus)

    Abstract

    Given a topic-by-run score matrix from past data, topic set size design methods can help test collection builders determine the number of topics to create for a new test collection from a statistical viewpoint. In this study, we apply a recently-proposed score standardisation method called std-AB to score matrices before applying topic set size design, and demonstrate its advantages. For topic set size design, std-AB suppresses score variances and thereby enables test collection builders to consider realistic choices of topic set sizes, and to handle unnormalised measures in the same way as normalised measures. In addition, even discrete measures that clearly violate normality assumptions look more continuous after applying std-AB, which may make them more suitable for statistically motivated topic set size design. Our experiments cover a variety of tasks and evaluation measures from NTCIR-12.

    Original languageEnglish
    Title of host publicationInformation Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings
    PublisherSpringer Verlag
    Pages16-28
    Number of pages13
    Volume9994 LNCS
    ISBN (Print)9783319480503
    DOIs
    Publication statusPublished - 2016
    Event12th Asia Information Retrieval Societies Conference, AIRS 2016 - Beijing, China
    Duration: 2016 Nov 302016 Dec 2

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume9994 LNCS
    ISSN (Print)03029743
    ISSN (Electronic)16113349

    Other

    Other12th Asia Information Retrieval Societies Conference, AIRS 2016
    CountryChina
    CityBeijing
    Period16/11/3016/12/2

    Fingerprint

    Standardization
    Violate
    Normality
    Design Method
    Design
    Cover
    Evaluation
    Experiments
    Demonstrate
    Experiment

    ASJC Scopus subject areas

    • Theoretical Computer Science
    • Computer Science(all)

    Cite this

    Sakai, T. (2016). The effect of score standardisation on topic set size design. In Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings (Vol. 9994 LNCS, pp. 16-28). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9994 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-48051-0_2

    The effect of score standardisation on topic set size design. / Sakai, Tetsuya.

    Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings. Vol. 9994 LNCS Springer Verlag, 2016. p. 16-28 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9994 LNCS).

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Sakai, T 2016, The effect of score standardisation on topic set size design. in Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings. vol. 9994 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9994 LNCS, Springer Verlag, pp. 16-28, 12th Asia Information Retrieval Societies Conference, AIRS 2016, Beijing, China, 16/11/30. https://doi.org/10.1007/978-3-319-48051-0_2
    Sakai T. The effect of score standardisation on topic set size design. In Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings. Vol. 9994 LNCS. Springer Verlag. 2016. p. 16-28. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-48051-0_2
    Sakai, Tetsuya. / The effect of score standardisation on topic set size design. Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings. Vol. 9994 LNCS Springer Verlag, 2016. pp. 16-28 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
    @inproceedings{4765a7231aae4535ba444c1a8ee40d1c,
    title = "The effect of score standardisation on topic set size design",
    abstract = "Given a topic-by-run score matrix from past data, topic set size design methods can help test collection builders determine the number of topics to create for a new test collection from a statistical viewpoint. In this study, we apply a recently-proposed score standardisation method called std-AB to score matrices before applying topic set size design, and demonstrate its advantages. For topic set size design, std-AB suppresses score variances and thereby enables test collection builders to consider realistic choices of topic set sizes, and to handle unnormalised measures in the same way as normalised measures. In addition, even discrete measures that clearly violate normality assumptions look more continuous after applying std-AB, which may make them more suitable for statistically motivated topic set size design. Our experiments cover a variety of tasks and evaluation measures from NTCIR-12.",
    author = "Tetsuya Sakai",
    year = "2016",
    doi = "10.1007/978-3-319-48051-0_2",
    language = "English",
    isbn = "9783319480503",
    volume = "9994 LNCS",
    series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
    publisher = "Springer Verlag",
    pages = "16--28",
    booktitle = "Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings",
    address = "Germany",

    }

    TY - GEN

    T1 - The effect of score standardisation on topic set size design

    AU - Sakai, Tetsuya

    PY - 2016

    Y1 - 2016

    N2 - Given a topic-by-run score matrix from past data, topic set size design methods can help test collection builders determine the number of topics to create for a new test collection from a statistical viewpoint. In this study, we apply a recently-proposed score standardisation method called std-AB to score matrices before applying topic set size design, and demonstrate its advantages. For topic set size design, std-AB suppresses score variances and thereby enables test collection builders to consider realistic choices of topic set sizes, and to handle unnormalised measures in the same way as normalised measures. In addition, even discrete measures that clearly violate normality assumptions look more continuous after applying std-AB, which may make them more suitable for statistically motivated topic set size design. Our experiments cover a variety of tasks and evaluation measures from NTCIR-12.

    AB - Given a topic-by-run score matrix from past data, topic set size design methods can help test collection builders determine the number of topics to create for a new test collection from a statistical viewpoint. In this study, we apply a recently-proposed score standardisation method called std-AB to score matrices before applying topic set size design, and demonstrate its advantages. For topic set size design, std-AB suppresses score variances and thereby enables test collection builders to consider realistic choices of topic set sizes, and to handle unnormalised measures in the same way as normalised measures. In addition, even discrete measures that clearly violate normality assumptions look more continuous after applying std-AB, which may make them more suitable for statistically motivated topic set size design. Our experiments cover a variety of tasks and evaluation measures from NTCIR-12.

    UR - http://www.scopus.com/inward/record.url?scp=85007082699&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85007082699&partnerID=8YFLogxK

    U2 - 10.1007/978-3-319-48051-0_2

    DO - 10.1007/978-3-319-48051-0_2

    M3 - Conference contribution

    SN - 9783319480503

    VL - 9994 LNCS

    T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

    SP - 16

    EP - 28

    BT - Information Retrieval Technology - 12th Asia Information Retrieval Societies Conference, AIRS 2016, Proceedings

    PB - Springer Verlag

    ER -