Throughout adulthood and ageing our brains undergo structural loss in an average pattern resembling faster atrophy in Alzheimer's disease (AD). Using a longitudinal adult lifespan sample (aged 30-89; 2-7 timepoints) and four polygenic scores for AD, we show that change in AD-sensitive brain features correlates with genetic AD-risk and memory decline in healthy adults. We first show genetic risk links with more brain loss than expected for age in early Braak regions, and find this extends beyond APOE genotype. Next, we run machine learning on AD-control data from the Alzheimer's Disease Neuroimaging Initiative using brain change trajectories conditioned on age, to identify AD-sensitive features and model their change in healthy adults. Genetic AD-risk linked with multivariate change across many AD-sensitive features, and we show most individuals over age ~50 are on an accelerated trajectory of brain loss in AD-sensitive regions. Finally, high genetic risk adults with elevated brain change showed more memory decline through adulthood, compared to high genetic risk adults with less brain change. Our findings suggest quantitative AD risk factors are detectable in healthy individuals, via a shared pattern of ageing- and AD-related neurodegeneration that occurs along a continuum and tracks memory decline through adulthood.