Profile-based focused Crawler for Social Media-Sharing Websites

In this paper, we present a novel profile based focused crawling system for dealing with increasingly popular social media-sharing web sites. In this system, we treat users' profiles as ranking criteria for guiding the crawling process. Furthermore, we divide a user's profile into two parts, an internal part which comes from the user's own contribution, and an external part, which comes from the user's social contacts. In order to efficiently and effectively extract data from a social media-sharing website for focused crawling, a path string based page-classification method was first developed for identifying list pages, detail pages and profile pages.