Evaluation with informational and navigational intents

Tetsuya Sakai*

*この研究の対応する著者

研究成果

43 被引用数 (Scopus)

抄録

Given an ambiguous or underspecified query, search result diversification aims at accomodating different user intents within a single "entry- point" result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as a simple method for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#- nDCG and P+Q#, are comparable to intent recall and D-nDCG, and possibly superior to α-nDCG; (b) In terms of preference agreement with intent recall, P+Q# is superior to other diversity metrics and therefore may be the most intuitive as a metric that emphasises diversity; and (c) In terms of preference agreement with effective precision, DIN#-nDCG is superior to other diversity metrics and therefore may be the most intuitive as a metric that emphasises relevance. Moreover, DIN#-nDCG may be the most intuitive as a metric that considers both diversity and relevance. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.

本文言語English
ホスト出版物のタイトルWWW'12 - Proceedings of the 21st Annual Conference on World Wide Web
ページ499-508
ページ数10
DOI
出版ステータスPublished - 2012 5 16
外部発表はい
イベント21st Annual Conference on World Wide Web, WWW'12 - Lyon, France
継続期間: 2012 4 162012 4 20

出版物シリーズ

名前WWW'12 - Proceedings of the 21st Annual Conference on World Wide Web

Conference

Conference21st Annual Conference on World Wide Web, WWW'12
国/地域France
CityLyon
Period12/4/1612/4/20

ASJC Scopus subject areas

  • コンピュータ ネットワークおよび通信

フィンガープリント

「Evaluation with informational and navigational intents」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル