Intelligent Peer Networks for Collaborative Web Search

Collaborative query routing is a new paradigm for web search that treats both established search engines and other publicly available indexes as intelligent peer agents in a search network. The approach makes it transparent for anyone to build his or her own (micro) search engine by integrating established web search services, desktop search, and topical crawling techniques. The challenge in this model is that each of these agents must team about its environment-the existence, knowledge, diversity, reliability and trustworthiness of other agents-by analyzing the queries received from and results exchanged with these other agents. We present the 6S peer network, which uses machine-learning techniques to learn about the changing query environment. We show that simple reinforcement learning algorithms are sufficient to detect and exploit semantic locality in the network, resulting in efficient routing and high-quality search results. A prototype of 6S is available for public use and is intended to assist in the evaluation of different AI techniques employed by the networked agents.