A method for indexing web pages using web bots

Exploring the content of web pages for automatic indexing is of fundamental importance for efficient e-commerce and other applications of the Web. It enables users, including customers and businesses, to locate the best sources for their use. Today's search engines use one of two approaches to indexing web pages. They either (i) analyze the frequency of the words (after filtering Out common or meaningless words) appearing in the entire or a part (typically, a title, an abstract or the first 300 words) of the text of the target web page, or (ii) they use sophisticated algorithms to take into account associations of words in the indexed web page. In both cases only words appearing in the web page in question are used in analysis. Often, to increase relevance of the selected terms to the potential searches, the indexing is refined by human processing. To identify so called "authority," or "expert" pages, some search engines use the structure of the links between pages to identify, pages that are often referenced by other pages. Analyzing the density, direction and clustering of links, this method is capable of identifying the pages that are likely to contain valuable information. It is analogous to a well known citation analysis method developed in library sciences and used by such publications as the Science Citation Index. A slightly different approach is used in the Google Search Engine implementation which assigns to each page a score that depends on frequency with which this page is visited by web surfers. The basic difference between the existing methods and the one discussed here is that these methods rely on a structure of web page linkages that lead from or to the indexed page. In contrast, our method uses the content of the pages linked to or from the indexed page for indexing. So our method uses a structure of words used by the linked pages, whereas the current methods use the structure of the connections between linked pages. In this paper we propose and demonstrate usage of a new method based on bots which analyze content of the pages linked to or from the page of interest. We analyze the similarity of the word usage at the different link distance from tile Page of interest and demonstrate that a structure of words used by the linked pages enables more efficient indexing and search.