Low-Resource Emotional Speech Synthesis: Transfer Learning and Data Requirements

Recently, a number of solutions were proposed that improved on ways of adding an emotional aspect to speech synthesis. Combined with core neural text-to-speech architectures that reach high naturalness scores, these models are capable of producing natural human-like speech with well discernible emotions and even model their intensities. To successfully synthesize emotions the models are trained on hours of emotional data. In practice however, it is often difficult and rather expensive to collect a lot of emotional speech data per speaker. In this article, we inquire upon the minimal data requirements of expressive text-to-speech solutions to be applied in practical scenarios and also find an optimal architecture for low-resource training. In particular, a different number of training speakers and a different amount of data per emotion are considered. Frequently occurring situations are considered when there is a large multi-speaker dataset with neutral records and a large single-speaker emotional dataset, but there is little emotional data for the remaining speakers. On top of that we study the effect of several architecture modifications and training procedures (namely adversarial training and transfer learning from speaker verification) on the quality of the models as well as their data avidity. Our results show that transfer learning may lower data requirements from 15 min per speaker per emotion to just 2.5-7 min maintaining non-significant changes in voice naturalness and giving high emotion recognition rates. We also show how the data requirements change from one emotion to another. A demo page illustrating the main findings of this work is available at: https://diparty.github.io/projects/tts/emo/nat.