A Deep Transfer Learning Framework for 3D Brain Imaging Based on Optimal Mass Transport

Deep learning has attracted increasing attention in brain imaging, but many neuroimaging data samples are small and fail to meet the training data requirements to optimize performance. In this study, we propose a deep transfer learning network based on Optimal Mass Transport (OMTNet) for 3D brain image classification using MRI scans from the UK Biobank. The major contributions of the OMTNet method include: a way to map 3D surface-based vertex-wise brain shape metrics, including cortical thickness, surface area, curvature, sulcal depth, and subcortical radial distance and surface Jacobian determinant metrics, onto 2D planar images for each MRI scan based on area-preserving mapping. Such that some popular 2D convolution neural networks pretrained on the ImageNet database, such as ResNet152 and DenseNet201, can be used for transfer learning of brain shape metrics. We used a score-fusion strategy to fuse all shape metrics and generate an ensemble classification. We tested the approach in a classification task conducted on 26k participants from the UK Biobank, using body mass index (BMI) thresholds as classification labels (normal vs. obese BMI). Ensemble classification accuracies of 72.8 +/- 1.2% and 73.9 +/- 2.3% were obtained for ResNet152 and DenseNet201 networks that used transfer learning, with 5.4-12.3% and 6.1-13.0% improvements relative to classifications based on single shape metrics, respectively. Transfer learning always outperformed direct learning and conventional linear support vector machines with 3.4-8.7% and 4.9-6.0% improvements in ensemble classification accuracies, respectively. Our proposed OMTNet method may offer a powerful transfer learning framework that can be extended to other vertex-wise brain structural/functional imaging measures.