Self-attention Based Text Matching Model with Generative Pre-training

Text matching is an important method to judge the semantic similarity of different sentences. Improving the efficiency and accuracy of text matching is the most focus in the field of information matching. In recent years, deep learning has been widely applied to text matching tasks and achieved good results. However, the different models have different limitations, such as CNN cannot learn global semantic information well, RNN cannot be parallelized well, and large pre-training language models have too many parameters to be deployed on hardware well. To address these problems, this paper propose a self-attention based text matching model with generative pre-training. Self-attention mechanism is adopted to learn the semantic information between words in a sentence, and can achieve better parallelization. We use the deep separable convolution model to obtain local features. In the pretraining stage of this model, a generative model variational autoencoder is used to learn the semantic relationship between similar sentences. And in the downstream text matching model, we employ Siamese Network structure, combine depth-wise separable convolutions and self-attention mechanism for feature extraction, and use attention mechanism for text interaction, in which the parameters in the pre-training phase will be shared. At last, we evaluate our model on three datasets: LCQMC, QQP, and a securities dataset. Experiment results show that our method achieves pretty good performance.