Weighted Rank Correlation in Information Retrieval Evaluation

In Information Retrieval (IR), it is common practice to compare the rankings observed during air experiment - the statistical procedure to compare rankings is called rank correlation. Rank correlation helps decide the success of new systems, models and techniques. To measure rank correlation, the most used coefficient; is Kendall's tau. However, in IR, when computing the correlations, the most relevant, useful or interesting items should often be considered more important than the least important items. Despite its simplicity and widespread use, Kendall's tau-little helps discriminate the items by importance. To overcome this drawback, in this paper, a family tau(*) of rank correlation coefficients for IR has been introduced for discriminating the rank correlation according to the rank of the items. The basis has been provided by the notion of gain previously utilized in retrieval effectiveness measurement. The probability distribution for tau(*) has also been provided.