Evaluating Server Selection for Federated Search

Previous evaluations of server selection methods for federated search I awe either used metrics which are unconnected with user satisfaction; or have not been able to account for confounding factors due to other search components. We propose a new framework for evaluating federated search server selection techniques. In our model; we isolate the effect of other confounding factors such as server summaries and result merging. Our results suggest that state-of-the-art server selection techniques are generally effective but result merging methods can be significantly improved. Furthermore, we show that the performance differences among server selection techniques can be obscured by ineffective merging.