To examine potential associations of the history of lifetime smoking and smoking cessation with cognitive function in the elderly. In a population-based cohort study of older adults in Saarland, Germany, a detailed lifetime history of smoking was obtained using standardised questionnaires. Cognitive function was assessed with a validated telephone-based instrument (COGTEL) at the five-year follow-up in a subsample of n = 1,697 participants with a baseline age >65 years. Multiple linear regression models were employed to predict cognitive performance, adjusting for potential confounding factors. Ever-smokers with a higher cumulative dose of smoking in pack-years scored lower in the cognitive assessment than never-smokers, with the association being more pronounced in current smokers than in former smokers. In fully adjusted models, current smokers with 21–40 pack-years scored 4.06 points lower (95 % CI −7.18 to −0.94) than never-smokers. In former smokers, a longer time since smoking cessation was associated with higher scores in the cognitive test with reference to current smokers, even after adjustment for pack-years. Former smokers who had quit for more than 30 years scored 4.23 points higher (95 % CI 1.75 to 6.71) than current smokers. Dose–response-relationships of cognitive function with cumulative dose of smoking as well as with time since smoking cessation were substantiated by restricted cubic splines regression. Our results support suggestions that smokers are at an increased risk for cognitive impairment in older age; that the risk increases with duration and intensity of smoking, and subsides with time after smoking cessation.