Semantic Vector Learning Using Pretrained Transformers in Natural Language Understanding

Natural language understanding (NLU) is a core technology for implementing natural interfaces. To implement and support robust NLU, previous studies introduced a neural network approach to learn semantic vector representation by employing the correspondence between text and semantic frame texts as extracted semantic knowledge. In their work, long short-term memory (LSTM)-based text and readers were used to encode both text and semantic frames. However, there exists significant room for performance improvement using recent pretrained transformer encoders. In the present work, as a key contribution, we have extended Jung’s framework to work with pretrained transformers for both text and semantic frame readers. In particular, a novel semantic frame processing method is proposed to directly feed the structural form of the semantic frame to transformers. We conducted massive experiments by combining various types of LSTM- or transformer-based text and semantic frame readers on the ATIS, SNIPS, Sim-M, Sim-R, and Weather datasets to find the best suitable configurations for learning effective semantic vector representations. Through the experiments, we concluded that the transformer-based text and semantic frame reader show a stable and rapid learning curve as well as the best performance in similarity-based intent classification and semantic search tasks. Copyright 2020. The Korean Institute of Information Scientists and Engineers