Learning Transferable 3D-CNN for MRI-Based Brain Disorder Classification from Scratch: An Empirical Study

Reliable and efficient transferability of 3D convolutional neural networks (3D-CNNs) is an important but extremely challenging issue in medical image analysis, due to small-sized samples and the domain shift problem (e.g., caused by the use of different scanners, protocols and/or subject populations in different sites/datasets). Although previous studies proposed to pretrain CNNs on ImageNet, models' transferability is usually limited due to semantic gap between natural and medical images. In this work, we try to answer a key question: how to learn transferable 3D-CNNs from scratch based on a small (e.g., tens or hundreds) medical image dataset? We focus on the case of structural MRI-based brain disorder classification using four benchmark datasets (i.e., ADNI-1, ADNI-2, ADNI-3 and AIBL) to address this problem. (1) We explore the influence of different network architectures on model transferability, and find that appropriately deepening or widening a network can increase the transferability (e.g., with improved sensitivity). (2) We analyze the contributions of different parts of 3D-CNNs to the transferability, and verify that fine-tuning CNNs can significantly enhance the transferability. This is different from the previous finding that fine-tuning CNNs (pretrained on ImageNet) cannot improve the model transferability in 2D medical image analysis. (3) We also study the between-task transferability when a model is trained on a source task from scratch and applied to a related target task. Experimental results show that, compared to directly training CNN on related target tasks, CNN pretrained on a source task can yield significantly better performance.