Background: The apolipoprotein E (APOE) e4 genetic polymorphism is a major risk factor for Alzheimer's disease, hence the possible prevention of its detrimental effects on cognition is of high relevance. Methods: We used linear regression models to assess associations of APOE e4 with cognitive performance in a population-based cohort study (n = 1,434) and in a cohort of patients with coronary heart disease (n = 366), and restricted cubic splines to explore dose-response relationships between serum cholesterol levels and cognition depending on APOE polymorphism. Results: The association of APOE e4 with cognitive function was strongly amplified in the presence of hypercholesterolemia and cardiovascular disease in both independent cohorts; hypercholesterolemia was associated with cognitive function only among APOE e4 carriers in the presence of cardiovascular disease. The interaction effect between APOE genotype and hypercholesterolemia was statistically significant in both cohorts. Conclusions: The detrimental effects of APOE e4 polymorphism on cognition may strongly depend on modifiable risk factors.