AlexSegNet: an accurate nuclei segmentation deep learning model in microscopic images for diagnosis of cancer

The nuclei segmentation of microscopic images is a key pre-requisite for cancerous pathological image analysis. However, an accurate nuclei cell segmentation is a long running major challenge due to the enormous color variability of staining, nuclei shapes, sizes, and clustering of overlapping cells. To address this challenges, we proposed a deep learning model, namely, AlexSegNet which is based upon AlexNet model Encoder-Decoder framework. In Encoder part, it stitches feature maps in the channel dimension to achieve feature fusion and uses a skip structure in Decoder part to combine low- and high-level features to ensure the segmentation effect of the nucleus. At final stage, we have also introduced a stacked network where feature maps are stacks on top of each other. We have used a publically available 2018 Data Science Bowl and Triple Negative Breast Cancer (TNBC) datasets of microscopic nuclei images for this study which comprises of several sample types such as small and large fluorescent, pink, purple, and grayscale tissue samples. Experimental results show that our proposed AlexSegNet achieved a segmentation maximum performance of 91.66% for Data Science Bowl dataset and 66.88% for TNBC dataset. The results are competitive compared to the results of other state-of-the-art models. This model is expected to be useful clinically for technician experts to succeed the analysis of cancer diagnosis into the survival chances of patients.