Refinement of Convolutional Neural Network Based Cell Nuclei Detection Using Bayesian Inference

Cytological samples provide useful data for cancer diagnostics but their visual analysis under a microscope is tedious and time-consuming. Moreover, some scientific tests indicate that various pathologists can classify the same sample differently or the same pathologist can classify the sample differently if there is a long interval between subsequent examinations. We can help pathologists by providing tools for automatic analysis of cellular structures. Unfortunately, cytological samples usually consist of clumped structures, so it is difficult to extract single cells to measure their morphometric parameters. To deal with this problem, we are proposing a nuclei detection approach, which combines convolutional neural network and Bayesian inference. The input image is preprocessed by the stain separation procedure to extract a blue dye (hematoxylin) which is mainly absorbed by nuclei. Next, a convolutional neural network is trained to provide a semantic segmentation of the image. Finally, the segmentation results are post processed in order to detect nuclei. To do that, we model the nuclei distribution on a plane using marked point process and apply the Besag's iterated conditional modes to find the configuration of ellipses that fit the nuclei distribution. Thanks to this we can represent clusters of occluded cell nuclei as a set of an overlapping ellipses. The accuracy of the proposed method was tested on 50 cytological images of breast cancer. Reference data was generated by the manual labeling of cell nuclei in images. The effectiveness of the proposed method was compared with the marker-controlled watershed. We applied our method and marker controlled watershed to detect nuclei in the semantic segmentation maps generated by the convolutional neural network. The accuracy of nuclei detection is measured as the number of true positive (TP) detections and false positive (FP) detections. It was recorded that the method can detect correctly 93.5% of nuclei (TP) and at the same time it generates only 6.1% of FR The proposed approach has led to better results than the marker-controlled watershed both in the number of correctly detected nuclei and in the number of false detections.