Many aspects of cognition decline from middle to late adulthood, but the dimensionality and generality of this decline have rarely been examined. We analyzed 20-year longitudinal data of 6203 middle-aged to very old adults from Greater Manchester and Newcastle-upon-Tyne, UK. Participants were assessed up to eight times on 20 tasks of fluid intelligence, perceptual speed, memory, and vocabulary. We controlled for potential effects due to retest, city, sex, and socio-economic class. Average performance in all tasks declined with age, and individual differences in decline were present for all but one memory and two vocabulary tasks. Half of the variance in level of performance was shared across tasks. This proportion increased to 66% for individual differences in change. General level of performance and change therein correlated positively. We conclude that cognitive decline is heterogeneous across individuals and rather general at the within-individual level.