The human brain changes with age, but the rate and the trajectories of change vary among the brain regions and among individuals, and the reasons for these differences are unclear. In a sample of healthy middle-aged and older adults, we examined mean volume change and individual differences in the rate of change in 12 regional brain volumes over approximately 30 months. In addition to the baseline assessment, there were two follow-ups, 15 months apart. We observed significant average shrinkage of the hippocampus, entorhinal cortex, orbital-frontal cortex, and cerebellum in each of the intervals. Shrinkage of the hippocampus accelerated with time, whereas shrinkage of the caudate nucleus, prefrontal subcortical white matter, and corpus callosum emerged only at the second followup. Throughout both assessment intervals, the mean volumes of the lateral prefrontal and primary visual cortices, putamen, and pons did not change. Significant individual differences in shrinkage rates were observed in the lateral prefrontal cortex, the cerebellum, and all the white matter regions throughout the study, whereas additional regions (medial-temporal structures, the insula, and the basal ganglia) showed significant individual variation in change during the second follow-up. No individual variability was noted in the change of orbital frontal and visual cortices. In two white matter regions, we were able to identify factors associated with individual differences in brain shrinkage. In corpus callosum, shrinkage rate was greater in persons with hypertension, and in the pons, women and carriers of the ApoEε4 allele exhibited declines not noted in the whole sample.