Uncertainty estimation for deep learning-based pectoral muscle segmentation via Monte Carlo dropout

Objective. Deep Learning models are often susceptible to failures after deployment. Knowing when your model is producing inadequate predictions is crucial. In this work, we investigate the utility of Monte Carlo (MC) dropout and the efficacy of the proposed uncertainty metric (UM) for flagging of unacceptable pectoral muscle segmentations in mammograms. Approach. Segmentation of pectoral muscle was performed with modified ResNet18 convolutional neural network. MC dropout layers were kept unlocked at inference time. For each mammogram, 50 pectoral muscle segmentations were generated. The mean was used to produce the final segmentation and the standard deviation was applied for the estimation of uncertainty. From each pectoral muscle uncertainty map, the overall UM was calculated. To validate the UM, a correlation between the dice similarity coefficient (DSC) and UM was used. The UM was first validated in a training set (200 mammograms) and finally tested in an independent dataset (300 mammograms). ROC-AUC analysis was performed to test the discriminatory power of the proposed UM for flagging unacceptable segmentations. Main results. The introduction of dropout layers in the model improved segmentation performance (DSC = 0.95 +/- 0.07 versus DSC = 0.93 +/- 0.10). Strong anti-correlation (r = -0.76, p < 0.001) between the proposed UM and DSC was observed. A high AUC of 0.98 (97% specificity at 100% sensitivity) was obtained for the discrimination of unacceptable segmentations. Qualitative inspection by the radiologist revealed that images with high UM are difficult to segment. Significance. The use of MC dropout at inference time in combination with the proposed UM enables flagging of unacceptable pectoral muscle segmentations from mammograms with excellent discriminatory power.