This study set out to examine the role of different adversities experienced at different life course stages on cognitive aging (i.e., level and change). Data from the longitudinal study: Survey of Health, Ageing, and Retirement in Europe (SHARE) with the selection of participants over 60 years were used (N = 2662, Mdn_age = 68, SD_age = 5.39) in a Structural Equation Modeling. In early life, the experience of hunger predicted lower delayed recall (β = - 0.10, p < 0.001) and verbal fluency (β = - 0.06, p = 0.001) performance in older age, whereas financial hardship predicted lower verbal fluency (β = - 0.06, p = 0.005) performance and steeper decline in delayed recall (β = - 0.11, p < 0.001). In early adulthood, financial hardship and stress predicted better delayed recall (financial hardship: β = 0.08, p = 0.001; stress: β = 0.07, p = 0.003) and verbal fluency performance (financial hardship: β = 0.08, p = 0.001; stress β = 0.10, p < 0.001), but no adversities were associated with a change in cognitive performance. In middle adulthood, no adversities were associated with the level of cognitive performance, but financial hardship predicted lower decline in delayed recall (β = 0.07, p = 0.048). This study highlights the importance of disentangling the period effect from the specific effect of the adversity experienced in the association between adversity and cognition in older age. Moreover, differential results for delayed recall and verbal fluency measures suggest that it is also important to consider the cognitive outcome domains examined.