A STUDY OF RANK-CONSTRAINED MULTILINGUAL DNNS FOR LOW-RESOURCE ASR

Multilingual Deep Neural Networks (DNNs) have been successfully used to exploit out-of-language data to improve under-resourced ASR. In this paper, we improve on a multilingual DNN by utilizing low-rank factorization (LRF) of weight matrices via Singular Value Decomposition (SVD) to sparsify a multilingual DNN. LRF was previously used for monolingual DNNs, yielding large computational savings without a significant loss in recognition accuracy. In this work, we show that properly applying LRF on a multilingual DNN can improve recognition accuracy for multiple low-resource ASR configurations. First, only the final weight layer is factorized. Since the output weight layer needs to be trained with language specific data, reducing the number of parameters is beneficial for under-resourced languages. It is common in multilingual DNN speech recognition, to further adapt the full neural network through retraining of the multilingual DNN on target language data. Again we observe that in low-resource situations, this adaptation can bring significant improvement if LRF is applied to all hidden layers. We demonstrate the positive effect of LRF in two very different scenarios: one is a phone recognition task for two related languages and the other is a word recognition task using five different languages from the GlobalPhone dataset.