Cross-sectional magnetic resonance imaging (MRI) studies reported significant associations between gray matter (GM) density changes in various limbic and neocortical areas and worst cognitive performances in elderly controls. Longitudinal studies in this field remain scarce and led to conflicting data. We report a clinico-radiological investigation of 380 cognitively preserved individuals who undergo neuropsychological assessment at baseline and after 18 months. All cases were assessed using a continuous cognitive score taking into account the global evolution of neuropsychological performances. The vast majority of Mini Mental State Examination (MMSE) 29 and 30 cases showed equal or worst performance at follow-up due to a ceiling effect. GM densities, white matter hyperintensities and arterial spin labeling (ASL) values were assessed in the hippocampus, amygdala, mesial temporal and parietal cortex at inclusion using 3 Tesla MRI Scans. Florbetapir positron emission tomography (PET) amyloid was available in a representative subsample of 64 cases. Regional amyloid uptake ratios (SUVr), mean cortical SUVr values (mcSUVr) and corresponding z-scores were calculated. Linear regression models were built to explore the association between the continuous cognitive score and imaging variables. The presence of an APOE-ε4 allele was negatively related to the continuous cognitive score. Among the areas studied, significant associations were found between GM densities in the hippocampus and amygdala but not mesial temporal and parietal areas and continuous cognitive score. Neither ASL values, Fazekas score nor mean and regional PET amyloid load was related to the cognitive score. In multivariate models, the presence of APOE-ε4 allele and GM densities in the hippocampus and amygdala were independently associated with worst cognitive evolution at follow-up. Our data support the idea that early GM damage in the hippocampus and amygdala occur long before the emergence of the very first signs of cognitive failure in brain aging.