Several cortical and sub-cortical regions in the right hemisphere, particularly in parietal and frontal lobe, but also in temporal lobe and thalamus, are part of neural networks critically implicated in spatial and attentional functions. Damage to different sites within these networks can cause hemispatial neglect. The aim of this study was to identify the neural substrates of different spatial processing components that are known to contribute to neglect symptoms. First, three different spatial tasks (visual search, bisection, and visual memory) were tested in 27 patients with focal right brain-damage. Voxel-based lesion–symptom mapping was used to determine the relationships between specific sites of damage and severity of deficits in these three spatial tasks. Secondly, fMRI was used in 26 healthy controls who performed the same tasks. In the healthy group, fMRI results showed a differential activation of regions within the parietal and frontal lobes during bisection and visual search, respectively. In the patients, we confirmed a critical role of right lateral parietal cortex in bisection, but lesions in frontal and temporal lobe were more critical for visual search. These data support the existence of distinct components in spatial attentional processes that might be damaged to different degrees in neglect patients.