Web search evaluation with informational and navigational intents

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single "entry-point" result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#-nDCG and P+Q#, are comparable to intent recall and D#-nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN#-nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D#-nDCG and DIN#-nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.

Original languageEnglish
Pages (from-to)145-155
Number of pages11
JournalJournal of Information Processing
Volume21
Issue number1
DOIs
Publication statusPublished - 2013
Externally publishedYes

Fingerprint

Websites
Testing

Keywords

  • Diversified search
  • Evaluation
  • Intents
  • Metrics
  • Search result diversification
  • Web search

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Web search evaluation with informational and navigational intents. / Sakai, Tetsuya.

In: Journal of Information Processing, Vol. 21, No. 1, 2013, p. 145-155.

Research output: Contribution to journalArticle

@article{2e9fcbc2a9a24c0a9eb28a7934a06de0,
title = "Web search evaluation with informational and navigational intents",
abstract = "Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single {"}entry-point{"} result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#-nDCG and P+Q#, are comparable to intent recall and D#-nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN#-nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D#-nDCG and DIN#-nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.",
keywords = "Diversified search, Evaluation, Intents, Metrics, Search result diversification, Web search",
author = "Tetsuya Sakai",
year = "2013",
doi = "10.2197/ipsjjip.21.145",
language = "English",
volume = "21",
pages = "145--155",
journal = "Journal of Information Processing",
issn = "0387-5806",
publisher = "Information Processing Society of Japan",
number = "1",

}

TY - JOUR

T1 - Web search evaluation with informational and navigational intents

AU - Sakai, Tetsuya

PY - 2013

Y1 - 2013

N2 - Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single "entry-point" result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#-nDCG and P+Q#, are comparable to intent recall and D#-nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN#-nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D#-nDCG and DIN#-nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.

AB - Given an ambiguous or underspecified web search query, search result diversification aims at accomodating different user intents within a single "entry-point" result page. However, some intents are informational, for which many relevant pages may help, while others are navigational, for which only one web page is required. We propose new evaluation metrics for search result diversification that considers this distinction, as well as the condordance test for comparing the intuitiveness of a given pair of metrics quantitatively. Our main experimental findings are: (a) In terms of discriminative power which reflects statistical reliability, the proposed metrics, DIN#-nDCG and P+Q#, are comparable to intent recall and D#-nDCG, and possibly superior to α-nDCG; (b) In terms of the concordance test which quantifies the agreement of a diversity metric with a gold standard metric that represents a basic desirable property, DIN#-nDCG is superior to other diversity metrics in its ability to reward both diversity and relevance at the same time. Moreover, both D#-nDCG and DIN#-nDCG significantly outperform α-nDCG in their ability to reward diversity, to reward relevance, and to reward both at the same time. In addition, we demonstrate that the randomised Tukey's Honestly Significant Differences test that takes the entire set of available runs into account is substantially more conservative than the paired bootstrap test that only considers one run pair at a time, and therefore recommend the former approach for significance testing when a set of runs is available for evaluation.

KW - Diversified search

KW - Evaluation

KW - Intents

KW - Metrics

KW - Search result diversification

KW - Web search

UR - http://www.scopus.com/inward/record.url?scp=84872458827&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84872458827&partnerID=8YFLogxK

U2 - 10.2197/ipsjjip.21.145

DO - 10.2197/ipsjjip.21.145

M3 - Article

AN - SCOPUS:84872458827

VL - 21

SP - 145

EP - 155

JO - Journal of Information Processing

JF - Journal of Information Processing

SN - 0387-5806

IS - 1

ER -