An Investigation of Positional Encoding in Transformer-based End-to-end Speech Recognition

In the Transformer architecture, the model does not intrinsically learn the ordering information of the input frames and tokens due to its self-attention mechanism. In sequence-to-sequence learning tasks, the missing of ordering information is explicitly filled up by the use of positional representation. Currently, there are two major ways of using positional representation: the absolute way and relative way. In both ways, the positional information is represented by positional vector. In this paper, we propose the use of positional matrix in the context of relative positional vector. Instead of adding the vectors to the key vectors in the self-attention layer, our method transforms the key vectors according to its position. Experiments on LibriSpeech dataset show that our approach outperforms the positional vector approach.