An Efficient Pairwise Comparison Scheme for Document Ranking

In this paper, an efficient annotation scheme based on pairwise comparison is proposed in order to evaluate document ranking algorithms in information retrieval systems. With the assumption that document relevance to query expressions abides the principle of transitivity, it is intended to devise a method that infers ranking order while keeping the number of paired comparisons at minimum. To this purpose, order-preserving dynamic Shannon-Fano trees are produced by estimating the probability of document insertion points through inter-document similarity measures. The performance of Shannon-Fano trees, which are based on retaining most likely insertion points at least possible heights, is compared with that of self-balancing binary trees. Incorporating them, the average values of the number of paired comparisons among randomly ordered documents retrieved from 10 predetermined query expressions are measured in an indexed collection of news crawled from the web. Moreover, it is also discussed as to which measures in the literature may be applied in order to evaluate document ranking techniques with respect to the reference ordering attained by pairwise comparisons.