Objective - The purpose of this study was to examine the relationship between fasting serum lipids and short-term verbal memory in middle-aged adults. Methods and Results - Total cholesterol, high-density lipoprotein cholesterol (HDL-C), triglycerides, and memory were measured twice, at mean ages 55 and 61, in 3673 male and female participants of the Whitehall II study. Short-term verbal memory was assessed using a 20-word list. Logistic regression was used to model associations between ATP-III categories of lipids and memory deficit ( recall of <= 4 words) and decline ( decrease of >= 2 words). Analyses were adjusted for education, occupational position, coronary heart disease, stroke, hypertension, use of medication, diabetes, smoking, and alcohol consumption. Compared to high HDL-C (>= 60 mg/dL), low HDL-C (<40 mg/dL) was associated with greater odds of memory deficit at the first ( OR = 1.27; 95% confidence interval [CI] = 0.91 to 1.77) and second wave of this study ( OR = 1.53; 95% CI = 1.04 to 2.25) in fully adjusted analysis. Decrease in HDL-C over the 5-year follow-up period was associated with decline in memory in the adjusted analysis ( OR = 1.61; 95% CI = 1.19 to 2.16); no interaction with APOE e4 status was present. Conclusions - HDL-C levels are potentially modifiable, and our results suggest that low HDL-C is associated with poor memory and decline in memory in middle-aged adults.