Joint speaker encoder and neural back-end model for fully end-to-end automatic speaker verification with multiple enrollment utterances

Conventional automatic speaker verification systems can usually be decomposed into a frontend model such as time delay neural network (TDNN) for extracting speaker embeddings and a back -end model such as statistics -based probabilistic linear discriminant analysis (PLDA) or neural network -based neural PLDA (NPLDA) for similarity scoring. However, the sequential optimization of the front-end and back -end models may lead to a local minimum, which theoretically prevents the whole system from achieving the best optimization. Although some methods have been proposed for jointly optimizing the two models, such as the generalized endto -end (GE2E) model and NPLDA E2E model, most of these methods have not fully investigated how to model the intra-relationship between multiple enrollment utterances. In this paper, we propose a new E2E joint method for speaker verification especially designed for the practical scenario of multiple enrollment utterances. To leverage the intra-relationship among multiple enrollment utterances, our model comes equipped with frame -level and utterance -level attention mechanisms. Additionally, focal loss is utilized to balance the importance of positive and negative samples within a mini -batch and focus on the difficult samples during the training process. We also utilize several data augmentation techniques, including conventional noise augmentation using MUSAN and RIRs datasets and a unique speaker embedding -level mixup strategy for better optimization.