A multi-scale data fusion framework for bone age assessment with convolutional neural networks

Bone age assessment (BAA) has various clinical applications such as diagnosis of endocrine disorders and prediction of final adult height for adolescents. Recent studies indicate that deep learning techniques have great potential in developing automated BAA methods with significant advantages over the conventional methods based on handcrafted features. In this paper, we propose a multi-scale data fusion framework for bone age assessment with X-ray images based on non-subsampled contourlet transform (NSCT) and convolutional neural networks (CNNs). Unlike the existing CNN-based BAA methods that adopt the original spatial domain image as network input directly, we pre-extract a rich set of features for the input image by performing NSCT to obtain its multi-scale and multi-direction representations. This feature pre-extraction strategy could be beneficial to network training as the number of annotated examples in the problem of BAA is typically quite limited. The obtained NSCT coefficient maps at each scale are fed into a convolutional network individually and the information from different scales are then merged to achieve the final prediction. Specifically, two CNN models with different data fusion strategies are presented for BAA: a regression model with feature-level fusion and a classification model with decision-level fusion. Experiments on the public BAA dataset Digital Hand Atlas demonstrate that the proposed method can obtain promising results and outperform many state-of-the-art BAA methods. In particular, the proposed approaches exhibit obvious advantages over the corresponding spatial domain approaches (generally with an improvement of more than 0.1 years on the mean absolute error), showing great potential in the future study of this field.