Spatio-temporal context based recurrent visual attention model for lymph node detection

False-positive reduction is one of the most crucial components in an automated lymph nodes (LNs) detection task in volumetric computed tomography (CT) scans, which is a highly sought goal for cancer diagnosis and early treatment. In this article, treating the three-dimensional (3D) LN detection task as object detection on sequence problem, we propose a novel spatio-temporal context-based recurrent visual attention model (STRAM) for the LNs false positive reduction. We firstly extract the deep spatial features maps for two-dimensional LN patches from pre-trained Inception-V3 model. A new Gaussian kernel-based spatial attention method is then presented to extract the most discriminating spatial features for the corresponding center slices. Additionally, to combine the temporal information between 3D CT slices, we devise a novel "Siamese" mixture density networks which can learn to adaptively focus on the most relevant parts of the CT slices. Considering the lesion areas always locate around the centroid of the 3D CT scans, a hard constraint is imposed on the predicted attention locations with batch normalization technique and the Siamese architecture. The proposed model is a fully differentiable unit that can be optimized end-to-end by using stochastic gradient descent. The effectiveness of our method is verified on LN dataset: 388 mediastinal LNs labeled by radiologists in 90 patient CT scans, and 595 abdominal LNs in 86 patient CT scans. Our method demonstrates sensitivities of about 87%/82% at 3 FP/vol. and 93%/89% at 6 FP/vol. for mediastinum and abdomen, respectively, which compares favorably to previous methods.