Unanimity-aware gain for highly subjective assessments

    研究成果: Conference article

    1 引用 (Scopus)

    抜粋

    IR tasks have diversified: human assessments of items such as social media posts can be highly subjective, in which case it becomes necessary to hire many assessors per item to reflect their diverse views. For example, the value of a tweet for a given purpose may be judged by (say) ten assessors, and their ratings could be summed up to define its gain value for computing a graded-relevance evaluation measure. In the present study, we propose a simple variant of this approach, which takes into account the fact that some items receive unanimous ratings while others are more controversial. We generate simulated ratings based on a real social-media-based IR task data to examine the effect of our unanimity-aware approach on the system ranking and on statistical significance. Our results show that incorporating unanimity can affect statistical significance test results even when its impact on the gain value is kept to a minimum. Moreover, since our simulated ratings do not consider the correlation present in the assessors' actual ratings, our experiments probably underestimate the effect of introducing unanimity into evaluation. Hence, if researchers accept that unanimous votes should be valued more highly than controversial ones, then our proposed approach may be worth incorporating.

    元の言語English
    ページ(範囲)39-42
    ページ数4
    ジャーナルCEUR Workshop Proceedings
    2008
    出版物ステータスPublished - 2017 1 1
    イベント8th International Workshop on Evaluating Information Access, EVIA 2017 - Tokyo, Japan
    継続期間: 2017 12 5 → …

    ASJC Scopus subject areas

    • Computer Science(all)

    フィンガープリント Unanimity-aware gain for highly subjective assessments' の研究トピックを掘り下げます。これらはともに一意のフィンガープリントを構成します。

  • これを引用