On user-oriented measurements of effectiveness of web information retrieval systems

Traditional performance measures of information retrieval systems include precision and recall and their variants. While these measures work well in closed-laboratory environments, they are not suitable for practical IR systems such as web search systems for two main reasons. One is that it is not practical to know all the relevant documents in the entire document collection, which is required for the recall measure. The other is that the precision-recall measure doesn't take the rankings of relevant documents directly into account. There are other measures that improve over the precision-recall measure, such as expected search length (ESL) and average search length (ASL). These measures are not intuitive and difficult to compute. We propose in this paper RankPower as an alternative measure for evaluating the effectiveness of IR systems such as a web search system. RankPower takes both the rank and the number of relevant documents in the returned list of documents in response to a given query into consideration. RankPower is bounded below as the number of relevant documents approaches infinity. Thus comparisons among different systems using RankPower become intuitive and easy.