Regional realness-aware generative adversarial networks for stain normalization

Stain normalization is the standardization of the color appearance and has been commonly used in computer-aided diagnosis (CAD) systems. Recently, Generative Adversarial Networks (GANs)-based methods are becoming popular for stain style transfer since they succeed in maintaining structural and color information. The CycleGAN, which is a variant of GANs, has been applied to stain normalization, showing state-of-the-art performance. CycleGAN uses PatchGAN as a discriminator that returns a matrix where each pixel represents the local region of the image. However, the generator utilizes the output of the PatchGAN only for network optimization. The gradient returning to the generator from discriminator is also quite small during training which causes optimizing the network parameters inefficiently. In this paper, we aim to extend the CycleGAN to handle this problem using the output of the convolution layer in the PatchGAN, which we call Regional Realness-Aware Mask, to guide the generator about which regions are incorrect in the input image. In this context, the generator pays more attention to these regions in the next iterations, producing more realistic images. The performance of the proposed Regional Realness-Aware Generative Adversarial Networks (RRAGAN) model was evaluated on a commonly used MITOS-ATYPIA histopathology dataset. The RRAGAN method achieved the highest results in stain normalization compared to the five state-of-the art methods with a large margin in terms of PSNR, SSIM, and RMSE metrics. It achieved PSNR of 24.109, SSIM of 0.933, and RMSE of 8.3576 under the same settings. We also evaluated the impact of the RRAGAN to the segmentation performance on the MICCAI'16 GlaS dataset. It improved the segmentation performance by 4.3% as it reduces the stain color variation. The proposed method could potentially be used as a preprocessing step and can significantly help CAD systems to demonstrate stable performance under color variations.