-
Department Info
-
-
Admissions
-
-
Academics
-
-
People
-
-
Research
-
-
-
- Browse reports by year:
- [ ALL 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 ]
University of Minnesota - Computer Science and Engineering Technical Report Abstract
Average Position of User Clicks as an Automated and Non-Intrusive Way of Evaluating Ranking Methods
Report Number: 03-009Date of Submission: 2/5/2003
Authors:
View Report:
PDF format
Abstract:
The need for an objective and automated way of evaluating the performance of different ranking/reranking methods is becoming increasingly important in the web search/meta search domain. There are various methods for ranking search results ranging from traditional information retrieval approaches to more recent methods based on link analysis and other quality measures that can be derived from the documents. There are also a number of strategies for combining different heuristics and answers from multiple experts. With all of these possibilities it is becoming increasingly difficult to find the best parameters, the best method, or the best mixture of methods that will maximize the quality for a particular query type or domain. This paper addresses the problem of automatically comparing the quality of the ordering of documents that are presented to the user as a sorted list according to believed relevance for a given topic or query. We introduce the average position of user clicks metric as an implicit, automated, and non-intrusive way of evaluating ranking methods. We also discuss under which situations and assumptions this metric can be used objectively by addressing various bias sources. Experiments performed in our meta search engine suggests that, this approach has the potential to sample a wide range of query types and users with greater statistical significance compared to methods that rely on explicit user judgements.