Contextualized Word Representations for Self-Attention Network

Transfer learning is one approach that could be used to better train deep neural networks. It plays a key role in initializing a network in computer vision applications as opposed to implementing a network from scratch which could he time-consuming. Natural Language Processing (NLP) shares a similar concept of transferring from large-scale data. Recent studies demonstrated that pretrained language models can be used to achieve state-of-the-art results on a multitude of NLP tasks such as sentiment analysis, machine translation and text summarization. In this paper, we demonstrate that a free RNN/CNN self attention model used for sentiment analysis can be improved with 2.53% by using contextualized word representation learned in a language modeling task.