Modality Classification and Concept Detection in Medical Images using Deep Transfer Learning

Medical image classification and concept detection are two important tasks for efficient and robust medical retrieval systems and also help with downstream tasks such as knowledge discovery, medical report generation, medical question answering, and clinical decision making. We investigate the effectiveness of transfer learning on the modality classification task using state-of-the-art deep convolutional neural networks pretrained on generic images. We also compare the performance of the traditional pipeline of handcrafted features with multi-label learning algorithms with end-to-end deep learning features for the concept detection task. Experimental results on the modality classification task show that transfer learning can leverage the patterns learned from large training data to the medical domain where little labeled data is available. Moreover, results on the concept detection task show that the deep learning approach provides better and more powerful feature representations compared to handcrafted feature extraction methods. The results on both tasks suggest that deep transfer learning methods are effective in the medical domain where data is scarce.