Learnable Topical Crawler through Online Semi-supervised Clustering

The performance of a traditional topical crawler heavily depends on the quality and comprehensiveness of the initial training samples. However, this is often impossible in real applications since preparing good initial training samples is difficult and time-consuming. It is ideal and appealing for a topical crawler if it can learn knowledge concerning the target topics from the ever-changing environment and adapt itself to these changes during successive crawling process. In this paper, we present a semi-supervised clustering method for building a learnable topical crawler. Our approach employs a constrained k-means clustering algorithm to detect new samples from crawled pages, which is fed to page classifier and link predictor for updating the learned models. This approach enables topical crawling systems with incremental learning capability and in turn improves crawling performance. Comparison experiments have been carried out between our approach and another traditional relevance score based sample generation approach. The experimental results have shown that our approach achieves better performance.