Survey of Open Source Natural Language Processing Tools

Natural language processing tools are functional integration components that realize various subtasks in the field of natural language processing, and provide effective support for text processing and text analysis. At present, there are many types of natural language processing tools, various tools have different levels of support for subtasks, and some tools are only suitable for some special text fields, which will cause confusion in the selection of tools. Firstly, according to the processing order, the subtasks supported by the tools are divided into auxiliary tasks, basic tasks and application tasks, and are introduced. 23 domestic and foreign natural language processing open source tools such as LTP, NLPIR and OpenNLP are selected, and the call methods and supported programming languages of these tools are compared to summarize the characteristics of various tools. Then, the implementation principles of various tool subtasks are divided into rule methods, statistical methods, neural network methods and combination methods for sorting and analysis, the shortcomings of current tools are discussed. Finally, the future development of natural language processing tools are prospected from the aspects of multimodal fusion, cognitive intelligence, model compression and efficient computing. © 2023 The Author(s).