A Survey of Abstractive Text Summarization Utilising Pretrained Language Models

We live in a digital era - an era of technology, artificial intelligence, big data, and information. The data and information on which we depend to fulfil several daily tasks and decision-making can become overwhelming to deal with and requires effective processing. This can be achieved by designing improved and robust automatic text summarization systems. These systems reduce the size of text document while retaining the salient information. The resurgence of deep learning and its progress from the Recurrent Neural Networks to deep transformer based PretrainedLanguage Models (PLM) with huge parameters and ample world and common-sense knowledge have opened the doors for huge success and improvement of theNatural Language Processing tasks including Abstractive Text Summarization (ATS). This work surveys the scientific literature to explore and analyze recent research on pre-trained language models and abstractive text summarization utilizing these models. The pretrained language models on abstractive summarization tasks have been analyzed quantitatively based on ROUGE scores on four standard datasets while the analysis of state-of-the-art ATS models has been conducted qualitatively to identify some issues and challenges encountered on finetuning large PLMs on downstream datasets for abstractive summarization. The survey further highlights some techniques that can help boost the performance of these systems. The findings in terms of performance improvement reveal that the models with better performance use either one or a combination of these strategies: (1) Domain Adaptation, (2) Model Augmentation, (3) Stable finetuning, and (4) Data Augmentation.