Mesial temporal lobe (MTL) is prominently affected in normal aging and associated with neurodegeneration in AD. Whether or not MTL atrophy is dependent on increasing amyloid load before the emergence of cognitive deficits is still disputed. We performed a 4.5-year longitudinal study in 75 older community dwellers (48 women, mean age: 79.3 years) including magnetic resonance imaging at baseline and follow-up, positron emission tomography amyloid during follow-up, neuropsychological assessment at 18 and 55 months, and APOE genotyping. Linear regression models were used to identify predictors of the MTL volume loss. Amyloid load was negatively associated with bilateral MTL volume at baseline explaining almost 10.5% of its variability. In multivariate models including time of follow-up and demographic variables (older age, male gender), this percentage exceeded 35%. The APOE4 allele independently contributed another 6%. Cognitive changes had a modest but still significant negative association with MTL volume loss. Our data support a multifactorial model including amyloid deposition, older age, male gender, APOE4 allele, and slight decline of cognitive abilities as independent predictors of MTL volume loss in brain aging.