On the reliability and intuitiveness of aggregated search metrics

Ke Zhou, Mounia Lalmas, Tetsuya Sakai, Ronan Cummins, Joemon M. Jose

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Citations (Scopus)

Abstract

Aggregating search results from a variety of diverse verticals such as news, images, videos and Wikipedia into a single interface is a popular web search presentation paradigm. Although several aggregated search (AS) metrics have been proposed to evaluate AS result pages, their properties remain poorly understood. In this paper, we compare the properties of existing AS metrics under the assumptions that (1) queries may have multiple preferred verticals; (2) the likelihood of each vertical preference is available; and (3) the topical relevance assessments of results returned from each vertical is available. We compare a wide range of AS metrics on two test collections. Our main criteria of comparison are (1) discriminative power, which represents the reliability of a metric in comparing the performance of systems, and (2) intuitiveness, which represents how well a metric captures the various key aspects to be measured (i.e. various aspects of a user's perception of AS result pages). Our study shows that the AS metrics that capture key AS components (e.g., vertical selection) have several advantages over other metrics. This work sheds new lights on the further developments and applications of AS metrics.

Original languageEnglish
Title of host publicationCIKM 2013 - Proceedings of the 22nd ACM International Conference on Information and Knowledge Management
Pages689-698
Number of pages10
DOIs
Publication statusPublished - 2013 Dec 11
Event22nd ACM International Conference on Information and Knowledge Management, CIKM 2013 - San Francisco, CA, United States
Duration: 2013 Oct 272013 Nov 1

Publication series

NameInternational Conference on Information and Knowledge Management, Proceedings

Conference

Conference22nd ACM International Conference on Information and Knowledge Management, CIKM 2013
CountryUnited States
CitySan Francisco, CA
Period13/10/2713/11/1

Keywords

  • Aggregated search
  • Discriminative power
  • Diversity
  • Evaluation
  • Intuitiveness
  • Metric
  • Reliability

ASJC Scopus subject areas

  • Decision Sciences(all)
  • Business, Management and Accounting(all)

Fingerprint Dive into the research topics of 'On the reliability and intuitiveness of aggregated search metrics'. Together they form a unique fingerprint.

  • Cite this

    Zhou, K., Lalmas, M., Sakai, T., Cummins, R., & Jose, J. M. (2013). On the reliability and intuitiveness of aggregated search metrics. In CIKM 2013 - Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (pp. 689-698). (International Conference on Information and Knowledge Management, Proceedings). https://doi.org/10.1145/2505515.2505691