Multimodal mixing convolutional neural network and transformer for Alzheimer's disease recognition

Early recognition of Alzheimer's disease (AD) and its precursor state, mild cognitive impairment (MCI), is pivotal in interrupting the progression of the disease and providing suitable treatment. Recent development in deep learning techniques has drawn great research attention for improving the efficacy of AD recognition. However, numerous current methods solely utilize data from a single auxiliary domain, limiting their ability to harness valuable intrinsic insights from multiple domains. To cope with the challenge, this paper is devoted to establishing an innovative multimodal medical data fusion model, termed as MMDF, to perform Alzheimer's disease recognition. Multimodal data including clinical records and medical images are used by the proposed approach, and backbone models are constructed using various data modalities. Specifically, a vision transformer model, which is termed as MRI_ViT, is tailored to recognize AD using brain magnetic resonance imaging (MRI) data. In parallel, a novel multi-scale attention-embedded one-dimensional (1D) convolutional neural network (MA-1DCNN) is devised for analyzing clinical records. Subsequently, these basic models are combined for creating a new data fusion model to recognize Alzheimer's disease. The experimental results reveal outstanding performance compared with state-of-the-art (SOTA) methods.