Spatial-Frequency Non-local Convolutional LSTM Network for pRCC Classification

The accurate classification of 3D medical images is a challenging task for current deep learning methods. Deep learning models struggle to extract features when the data size is small and the data dimension is large. To solve this problem, we develop a spatial-frequency non-local convolutional LSTM network for 3D image classification. Compared to traditional networks, the proposed model has the ability to extract features from both the spatial and frequency domains, which allows the frequency-domain features to contribute to the classification. Furthermore, the non-local blocks in our architecture enable it to capture the long-range dependencies directly in the feature space. Finally, to simplify the classification task and improve the performance, we utilize a two-stage framework that localizes lesions in the first step, and classifies them in the second. We evaluate our method on a challenging and important clinical task, i.e, the differentiation of papillary renal cell carcinoma (pRCC) into subtype 1 and subtype 2. To the best of our knowledge, this is the first time that the advantage of synthesizing spatial- and frequency-domain features by deep learning networks for medical image classification has been demonstrated. Experimental results demonstrate that the proposed method achieves competitive and often superior performance compared to state-of-the-art networks and three clinical experts.