Joint segmentation and classification of skin lesions via a multi-task learning convolutional neural network

Skin lesion segmentation and classification are two crucial and correlated tasks in computer-aided diagnosis of skin diseases. Jointly performing these two tasks can exploit their correlations to obtain performance gains, but it remains a challenging topic. In this paper, we propose an end-to-end multi-task learning convolutional neural network (MTL-CNN) for joint skin lesion segmentation and classification, and additionally introduce edge prediction as an auxiliary task. Overall, MTL-CNN includes a shared encoder, two parallel decoders for generating edge and segmentation masks, and a classification subnet. First, the shared encoder is used to extract features for three tasks (i.e., edge prediction, segmentation, and classification). Then, we propose two kinds of simple but efficient modules to exploit the benefits among these three tasks. Specifically, we design multiple edge information enhancement (EIE) modules between the encoder and the segmentation decoder, aiming at introducing the edge information from the edge decoder as strong cues to enhance the edge parts of the segmentation features. These enhanced segmentation features are sent to the segmentation decoder for better segmentation. Besides, we design multiple lesion area extraction (LAE) modules between the encoder and the classification subnet, which aim to utilize the segmentation results to filter out the distractions on the classification features. These filtered classification features are input to the classification subnet and progressively fused in a bottom-up manner for classification. A three-phase training strategy is employed to train MTL-CNN. Extensive experiments on three datasets demonstrate the superiority of MTL-CNN over state-of-the-art segmentation, classification, and other multi-task approaches.