Recent evidence documents that grandparents raising grandchildren face a great risk of psychological distress associated with this caregiver role. Information we know about grandparent caregivers heavily relies on grandmothers. Although the caregiver's gender is an important contextual factor in shaping the stress process, often unnoticed are grandfathers who assume the caregiving responsibility for their grandchildren. Guided by the Caregiver Stress Process model, this study examines common and differential factors associated with the psychological health of grandfathers and grandmothers living with their grandchildren without the parents of the child. Based on the National Survey of America's Families, data on 209 grandfathers and 1,781 grandmothers were analyzed. Results indicate that factors related to a lower level of the psychological health of grandmothers include their younger age, poor health, not having high school diploma, not being employed, living in poverty, and caring for grandchildren with severe behavioral problems. On the contrary, factors associated with the poor psychological health of grandfathers include their living without a spouse, poor health, and caring for very young grandchildren (under the age of five) or grandchildren with severe behavioral problems. Active social participation (volunteer activities and religious services attendance) is directly associated with better psychological health only for grandmothers. Findings suggest that social policy and interventions to help grandparents raising grandchildren should be designed in consideration of the differential factors associated with mental health by the gender of caregiving grandparents. Implications for policy and practice are discussed.