Weakly Supervised Classification for Nasopharyngeal Carcinoma with Transformer in Whole Slide Images

Pathological examination of nasopharyngeal carcinoma (NPC) is an indispensable factor for diagnosis, guiding clinical treatment and judging prognosis. Traditional and fully supervised NPC diagnosis algorithms require manual delineation of regions of interest on the gigapixel of whole slide images (WSIs), which however is laborious and often biased. In this paper, we propose a weakly supervised framework based on Tokens-to-Token Vision Transformer (WS-T2T-ViT) for accurate NPC classification with only a slide-level label. The label of tile images is inherited from their slide-level label. Specifically, WS-T2T-ViT is composed of the multi-resolution pyramid, T2T-ViT and multi-scale attention module. The multi-resolution pyramid is designed for imitating the coarse-to-fine process of manual pathological analysis to learn features from different magnification levels. The T2T module captures the local and global features to overcome the lack of global information. The multi-scale attention module improves classification performance by weighting the contributions of different granularity levels. Extensive experiments are performed on the 802-patient NPC and CAMELYON16 dataset. WS-T2T-ViT achieves an area under the receiver operating characteristic curve (AUC) of 0.989 for NPC classification on the NPC dataset. The experiment results of CAMELYON16 dataset demonstrate the robustness and generalizability of WS-T2T-ViT in WSI-level classification. © 2013 IEEE.