Brain atrophy is a key factor behind episodic memory loss in aging, but the nature and ubiquity of this relationship remains poorly understood. This study leverages 13 longitudinal datasets, including 3737 cognitively healthy adults (10,343 MRI scans; 13,460 memory assessments), to determine whether brain change-memory change associations are more pronounced with age and genetic risk for Alzheimer’s Disease. Both factors are associated with accelerated brain decline, yet it remains unclear whether memory loss is exacerbated beyond what atrophy alone would predict. Additionally, we assess whether memory decline aligns with a global pattern of atrophy or stems from distinct regional contributions. Our mega-analysis reveals a nonlinear relationship between memory decline and brain atrophy, primarily affecting individuals with above-average brain structural decline. The associations are stronger in the hippocampus but also spread across diverse cortical and subcortical regions. The associations strengthen with age, reaching moderate associations in participants in their eighties. While APOE ε4 carriers exhibit steeper brain and memory loss, genetic risk has no effect on the change-change associations. These findings support the presence of common biological macrostructural substrates underlying memory function in older age which are vulnerable to multiple age-related factors, even in the absence of overt pathological changes.