This study aimed to examine changes in physical activity (PA) over time (2009–2017) in the same participants and to determine an association between changes in PA and health-related quality of life (HRQoL) in early older adults (n = 994) using data from the Korea Health Panel Survey. HRQoL was measured using the EuroQol system, and the amount of PA was grouped into four activity levels: remained inactive, became inactive, became active, and remained active. The association of changes in PA over 8 years with HRQoL was examined using logistic regression analysis while controlling for socioeconomic and behavioral factors. Total PA decreased from 1859.72 ± 1760.01 MET-minutes in 2009 to 1264.80 ± 1251.14 MET-minutes in 2017 (P < 0.001). In 2017, 142 (14.3%) remained inactive, whereas 419 (42.2%) remained active. Participants who remained inactive in early old age were more likely to be in the lowest 10% HRQoL of the sample (odds ratio = 1.95, 95% confidence interval = 1.09–3.48). These findings indicate that health education and promotion must be prioritized for middle-aged adults, who are relatively inactive, so that they increase their current PA and improve their HRQoL to maximize the benefits of PA in old age.