Evaluating the Summarization Comprehension of Pre-Trained Language Models

Recent advances in abstractive summarization demonstrate the importance of pre-training tasks, however, general purpose language models manage to outperform summarization-specialized pre-training approaches. While several works addressed the question of pseudosummarization pre-training efficiency in abstractive summarization fine-tuning, none has explored the properties of pre-trained models in a low-resource setting. This work attempts to fill this gap. We benchmark 5 state-of-the-art pre-trained language models on 5 single-document abstractive summarization datasets of different domains. To probe the models, we propose 4 novel task comprehension tests that evaluate themain components of summarizationmodels. Our experiments reveal that pseudo-summarization pre-training biases the models towards more extractive behavior and inhibits their ability to properly filter the salient content, leading to worse generalization.