Topic set size design with the evaluation measures for short text conversation

Tetsuya Sakai, Lifeng Shang, Zhengdong Lu, Hang Li

    研究成果: Conference contribution

    3 引用 (Scopus)

    抜粋

    Short Text Conversation (STC) is a new NTCIR task which tackles the following research question: given a microblog repository and a new post to that microblog, can systems reuse an old comment from the respository to satisfy the author of the new post? The official evaluation measures of STC are normalised gain at 1 (nG@1), normalised expected reciprocal rank at 10 (nERR@10), and P+, all of which can be regarded as evaluation measures for navigational intents. In this study, we apply the topic set size design technique of Sakai to decide on the number of test topics, using variance estimates of the above evaluation measures. Our main conclusion is to create 100 test topics, but what distinguishes our work from other tasks with similar topic set sizes is that we know what this topic set size means from a statistical viewpoint for each of our evaluation measures. We also demonstrate that, under the same set of statistical requirements, the topic set sizes required by nERR@10 and P+ are more or less the same, while nG@1 requires more than twice as many topics. To our knowledge, our task is the first among all efforts at TREC-like evaluation conferences to actually create a new test collection by using this principled approach.

    元の言語English
    ホスト出版物のタイトルLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    出版者Springer Verlag
    ページ319-331
    ページ数13
    9460
    ISBN(印刷物)9783319289397
    DOI
    出版物ステータスPublished - 2015
    イベント11th Asia Information Retrieval Societies Conference, AIRS 2015 - Brisbane, Australia
    継続期間: 2015 12 22015 12 4

    出版物シリーズ

    名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    9460
    ISSN(印刷物)03029743
    ISSN(電子版)16113349

    Other

    Other11th Asia Information Retrieval Societies Conference, AIRS 2015
    Australia
    Brisbane
    期間15/12/215/12/4

      フィンガープリント

    ASJC Scopus subject areas

    • Computer Science(all)
    • Theoretical Computer Science

    これを引用

    Sakai, T., Shang, L., Lu, Z., & Li, H. (2015). Topic set size design with the evaluation measures for short text conversation. : Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (巻 9460, pp. 319-331). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 巻数 9460). Springer Verlag. https://doi.org/10.1007/978-3-319-28940-3_25