A Chinese word segmentation model for energy literature based on Conditional Random Fields

Chinese word segmentation is one of the foundation and core tasks for Chinese natural language processing. Although some achievements have been made for Chinese word segmentation system in general domains, it is far away to meet practical requirements in energy domain. We focus on Chinese word segmentation standard and segmentation technology in the energy domain which consists of 13283 energy basic terms. This paper firstly proposes a conditional random field segmentation model. Then, the character features, character type features and conditional entropy features which influence the word segmentation performance are chose and described. Finally, the proposed model is tested on the dataset of the State Grid energy literature and compared with current word segmentation tools, such as the Harbin Institute of Technology's Language Technology Platform and the Tsinghua's THU Lexical Analyzer for Chinese language processing tools. The F1 value of the best result of the proposed model is 0.8319.