The mentalizing network (MN) treats social interactions based on our understanding of other people's intentions and includes the medial prefrontal cortex (mPFC), temporoparietal junction (TPJ), posterior cingulate cortex (PCC), precuneus (PC), and amygdala. Not all elders are equally affected by the aging-related decrease of mentalizing abilities. Personality has recently emerged as a strong determinant of functional connectivity in MN areas. However, its impact on volumetric changes across the MN in brain aging is still unknown. To address this issue, we explored the determinants of volume decrease in MN components including amyloid burden, personality, and APOE genotyping in a previously established cohort of 130 healthy elders with a mean follow-up of 54 months. Personality was assessed with the Neuroticism Extraversion Openness Personality Inventory-Revised. Regression models corrected for multiple comparisons were used to identify predictors of volume loss including time, age, sex, personality, amyloid load, presence of APOE epsilon 4 allele, and cognitive evolution. In cases with higher Agreeableness scores, there were lower volume losses in PCC, PC, and amygdala bilaterally. This was also the case for the right mPFC in elders displaying lower Agreeableness and Conscientiousness. In multiple regression models, the effect of Agreeableness was still observed in left PC and right amygdala and that of Conscientiousness was still observed in right mPFC volume loss (26.3% of variability, significant age and sex). Several Agreeableness (Modesty) and Conscientiousness (order, dutifulness, achievement striving, and self-discipline) facets were positively related to increased volume loss in cortical components of the MN. In conclusion, these data challenge the beneficial role of higher levels of Agreeableness and Conscientiousness in old age, showing that they are associated with an increased rate of volume loss within the MN.