A scalable and efficient probabilistic information retrieval and text mining system

A system for probabilistic information retrieval and text mining that is both scalable and efficient is presented. Separate feature extraction or stop-word lists are not needed since the system can remove unneeded paxameters dynamically based on a local mutual information measure. This is shown to be as effective as using a global measure. A novel way of storing system parameters eliminates the need for a ranking step during information retrieval from queries. Probability models over word contexts provide a method to suggest related words that can be added to a query. Test results are presented on a categorization task and screen shots from a live system axe shown to demonstrate its capabilities.