Cognitive aging is characterized by large heterogeneity, which may be due to variations in childhood socioeconomic conditions (CSC). Although there is substantial evidence for an effect of CSC on levels of cognitive functioning at older age, results on associations with cognitive decline are mixed. We examined by means of an accelerated longitudinal design the association between CSC and cognitive trajectories from 50 to 96 years. Cognition included two functions generally found to decline with aging: delayed recall and verbal fluency. Data are from six waves of the Survey of Health, Aging, and Retirement in Europe (SHARE), conducted between 2004 and 2015 (n = 24,066 at baseline; 56% female, age 50+). We found a consistent CSC pattern in levels of cognitive functioning in later life. Older people with disadvantaged CSC had lower levels of cognitive functioning than those with more advantaged CSC. We also find that decline is almost 1.6 times faster in the most advantaged group compared with the most disadvantaged group. The faster decline for people with more advantaged CSC becomes less pronounced when we additionally control for adulthood socioeconomic conditions and current levels of physical activity, depressive symptoms, and partner status. Our findings are in line with the latency, pathway, and cumulative model and lend support to theories of cognitive reserve, stating that neuronal loss can no longer be repaired in people with more cognitive reserve once the underlying pathology is substantial and speed of decline is accelerated.