Objective: This study aims to evaluate the impact of social isolation and loneliness, individually and simultaneously, on cognitive function in older adults during a 4-year period, using data from the English Longitudinal Study of Ageing, and to evaluate if these associations are moderated by educational level. Methods: Data on social isolation, loneliness, and cognitive function (verbal fluency, immediate recall, and delayed recall) were obtained at baseline. Follow-up measures on cognitive function were obtained 4 years later for 6034 participants (mean age at baseline = 65.6 years). Regression analyses were used to evaluate the association between baseline isolation, loneliness, and cognitive function at follow-up. Interactions between social isolation, loneliness, and educational level were also evaluated. Results: Baseline isolation was significantly associated with decreases in all cognitive function measures at follow-up (beta = -.05 to -.03, p < .001), independently of baseline scores, whereas loneliness was associated with poorer immediate recall (beta = -.05, p < .001) and delayed recall (beta = -.03, p = .02). There was a significant interaction between educational level and both isolation (p = .02) and loneliness (p = .01) for delayed recall, such that isolation and loneliness were associated with poorer recall only among those with low levels of education. Conclusions: Loneliness and isolation are associated with poorer cognitive function among older adults. Interventions to foster social connections may be particularly beneficial for individuals with low levels of education.