We explored whether chronic mood disorder has a long-term «toxic effect» on cognition and brain structures. We performed a 2-year longitudinal study of four groups of elderly individuals: euthymic patients suffering from bipolar disorder (BD), euthymic patients presenting an early (EOD) or a late-onset (LOD) of depression and healthy elderly. Regarding BD, patients displayed significantly lower performances in processing speed and episodic memory but not in working memory and executive functions compared to controls. However, BD patients did not differ from controls in the mean trajectory of cognitive changes during the two years follow-up. In the same line, longitudinal grey matter and white matter changes did not differ between BD patients and controls. Our results suggest that BD does not have a significant adverse impact. Regarding EOD, results revealed that cognitive performances as well as brain volumes were preserved in EOD patients both at baseline and at follow-up. LOD patients showed decreased performances limited to episodic memory tasks. Besides, MRI analyses found that they had significantly greater periventricular hyperintensities than controls. Our results support the distinction between the aetiological mechanisms implicated in the pathogenesis of LOD and EOD.