Towards automatic evaluation of multi-turn dialogues: A task design that leverages inherently subjective annotations

Tetsuya Sakai*

*この研究の対応する著者

    研究成果: Conference article査読

    1 被引用数 (Scopus)

    抄録

    This paper proposes a design of a shared task whose ultimate goal is automatic evaluation of multi-turn, dyadic, textual helpdesk dialogues. The proposed task takes the form of an offline evaluation, where participating systems are given a dialogue as input, and output at least one of the following: (1) an estimated distribution of the annotators' quality ratings for that dialogue; and (2) an estimated distribution of the annotators' nugget type labels for each utterance block (i.e., a maximal sequence of consecutive posts by the same utterer) in that dialogue. This shared task should help researchers build automatic helpdesk dialogue systems that respond appropriately to inquiries by considering the diverse views of customers. The proposed task has been accepted as part of the NTCIR-14 Short Text Conversation (STC-3) task. While estimated and gold distributions are traditionally compared by means of root mean squared error, Jensen-Shannon divergence and the like, we propose a pilot measure that considers the order of the probability bins for the dialogue quality subtask, which we call Symmetric Normalised Order-aware Divergence (SNOD).

    本文言語English
    ページ(範囲)24-30
    ページ数7
    ジャーナルCEUR Workshop Proceedings
    2008
    出版ステータスPublished - 2017 1 1
    イベント8th International Workshop on Evaluating Information Access, EVIA 2017 - Tokyo, Japan
    継続期間: 2017 12 5 → …

    ASJC Scopus subject areas

    • コンピュータ サイエンス(全般)

    フィンガープリント

    「Towards automatic evaluation of multi-turn dialogues: A task design that leverages inherently subjective annotations」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

    引用スタイル