On the reliability and intuitiveness of aggregated search metrics

Ke Zhou, Mounia Lalmas, Tetsuya Sakai, Ronan Cummins, Joemon M. Jose

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    7 Citations (Scopus)

    Abstract

    Aggregating search results from a variety of diverse verticals such as news, images, videos and Wikipedia into a single interface is a popular web search presentation paradigm. Although several aggregated search (AS) metrics have been proposed to evaluate AS result pages, their properties remain poorly understood. In this paper, we compare the properties of existing AS metrics under the assumptions that (1) queries may have multiple preferred verticals; (2) the likelihood of each vertical preference is available; and (3) the topical relevance assessments of results returned from each vertical is available. We compare a wide range of AS metrics on two test collections. Our main criteria of comparison are (1) discriminative power, which represents the reliability of a metric in comparing the performance of systems, and (2) intuitiveness, which represents how well a metric captures the various key aspects to be measured (i.e. various aspects of a user's perception of AS result pages). Our study shows that the AS metrics that capture key AS components (e.g., vertical selection) have several advantages over other metrics. This work sheds new lights on the further developments and applications of AS metrics.

    Original languageEnglish
    Title of host publicationInternational Conference on Information and Knowledge Management, Proceedings
    Pages689-698
    Number of pages10
    DOIs
    Publication statusPublished - 2013
    Event22nd ACM International Conference on Information and Knowledge Management, CIKM 2013 - San Francisco, CA
    Duration: 2013 Oct 272013 Nov 1

    Other

    Other22nd ACM International Conference on Information and Knowledge Management, CIKM 2013
    CitySan Francisco, CA
    Period13/10/2713/11/1

      Fingerprint

    Keywords

    • Aggregated search
    • Discriminative power
    • Diversity
    • Evaluation
    • Intuitiveness
    • Metric
    • Reliability

    ASJC Scopus subject areas

    • Business, Management and Accounting(all)
    • Decision Sciences(all)

    Cite this

    Zhou, K., Lalmas, M., Sakai, T., Cummins, R., & Jose, J. M. (2013). On the reliability and intuitiveness of aggregated search metrics. In International Conference on Information and Knowledge Management, Proceedings (pp. 689-698) https://doi.org/10.1145/2505515.2505691