An improved Dijkstra algorithm in Chinese Word Segmentation

This paper presents a pragmatic approach to Chinese word segmentation. It applies an improved Dijkstra algorithm to segment Chinese character stream. This research uses three steps to process an input Chinese character stream. First, find all possible Chinese word candidates based on the dictionary. Second, apply an improved Dijkstra's APSP theory to find a word path which costs the least fee. It is accomplished by computing all these word candidates' best left neighbor and their aggregate fee. During this procedure, the key step is to find the last candidate word which has the minimum aggregate fee from all last possible candidate words. Third, find all continuous single-character candidate words in the result of step two to check if they can be an entity name. The third step is the main difference from most of other previous works; it does improve the efficiency of recognizing entity names and also improve the precision of recognizing new words. This approach has been implemented in a search engine demo system based on Lucene. It consists of lexicon word processing and OOV (out of lexicon words) recognition. Experiments show that the algorithm does work well.