An integrated distributed system for web news retrieval

This paper highlights the problems of information explosion and the incapability of currently available search engines in finding what we mostly want. In particularly, these search engines cannot offer users the facility of specifying the categories and time frames they receive and cannot provide the online news information with the required frequency. To address these problems, we present the design and implementation of- "Ai-Times", a distributed web news retrieval system which can accurately retrieve and organize the web news information. We describe the optimized crawler algorithm, the news extraction algorithm, and explain how MapReduce is used in "Ai-Times" and can be improved to get better performance.