Elderly individuals display a rapid age-related increase in intraindividual variability (IIV) of their performances. This phenomenon could reflect subtle changes in frontal lobe integrity. However, structural studies in this field are still missing. To address this issue, we computed an IIV index for a simple reaction time (RT) task and performed magnetic resonance imaging (MRI) including voxel based morphometry (VBM) and the tract based spatial statistics (TBSS) analysis of diffusion tensor imaging (DTI) in 61 adults aged from 22 to 88 years. The age-related IIV increase was associated with decreased fractional anisotropy (FA) as well as increased radial (RD) and mean (MD) diffusion in the main white matter (WM) fiber tracts. In contrast, axial diffusion (AD) and grey matter (GM) densities did not show any significant correlation with IIV. In multivariate models, only FA has an age-independent effect on IIV. These results revealed that WM but not GM changes partly mediated the age-related increase of IIV. They also revealed that the association between WM and IIV could not be only attributed to the damage of frontal lobe circuits but concerned the majority of interhemispheric and intrahemispheric corticocortical connections.