Research of text segmentation based on parallel genetic algorithm

Focusing on the data sparseness of short texts, an algorithm based on knowledge from external corpus is proposed to improve the accuracy of text segmentation, which contains two steps: Gibbs sampling is adopted to estimate the LDA model; corresponding to the corpus and the latent semantic structure information of the text is inferred based on the LDA model. Two objective functions of internal cohesion and external dissimilarity are then defined to transform text segmentation into a multi-objective optimization problem. A parallel genetic algorithm based on the objective functions is employed to obtain the global optimal solution for text segmentation. According to the experiments, the proposed algorithm achieves higher accuracy than the MDA and LDA-based methods in the case of data sparseness.