Alzheimer's disease (AD) is widely aware as a neurodegenerative disease that is characterized as a leading cause of irreversible progressive dementia. From a clinical perspective, it is vital to forecast a patient's progression over time. In that regard, there have been rigorous researches on AD progression modeling with structural magnetic resonance imaging (MRI). Methodologically, there are three major aspects of MRI modeling: (i) variability over time, (ii) sparseness in observations, and (iii) geometrical properties in temporal dynamics. While the existing deep-learning-based methods have addressed variability or sparsity in data, there is still a need to take into account the inherent geometrical properties. Recently, geometric modeling based on ordinary differential equations (ODE-RGRU) has shown its ability in various time-series data by combining an RNN and an ordinary differential equation (ODE) in symmetric positive definite (SPD) space. Despite the success of ODE-RGRU in timeseries data modeling, it is limited to estimating the SPD matrix from sparse data with missing values. To this end, we propose a novel geometric learning framework for AD progression modeling to tackle the aforementioned issues simultaneously. And, we also propose training algorithms for manifold mapping on irregular and incomplete MRI and cognitive scores observations. Our proposed framework efficiently learns three major aspects of longitudinal MRI biomarker and cognitive scores by the manifold transformation module, ODE-RGRU, and missing value estimation module. We demonstrate the effectiveness of our method in experiments that forecast multi-class classification and cognitive scores over time. Additionally, we provide a multi-faceted analysis of the proposed method through an ablation study.