Creating the ultimate research assistant

A text-mining technology is any software application that derives structure and content from unstructured text and performs concept extraction, text summarization, and automated taxonomy generation. This technology can be combined with commercial Internet search engines to boostrap the creation of knowledge bases, encyclopedias, topic maps, and other knowledge organization systems, thus creating the ultimate research assistant. Using text-mining tools, an ISS knowledge base can be autogenerated from a Google search result set in 15 minutes or less. A custom tool that leverages the Google Web search API accomplishes various tasks using four main components. First, a search engine client discovers a list of relevant Web pages using the Goggle Web search API. An information extraction engine then mines concepts and associated text passages from these Web pages. Next, a clustering engine organizes the most significant concepts into a hierarchical taxonomy. Finally, a knowledge base generator uses this taxonomy to generate a hypertext knowledge base from the extracted concepts and text passages.