Prism adaptation is a well-known method used to investigate brain plasticity, and a promising technique for the rehabilitation of unilateral spatial neglect (USN). Only little evidence about the mechanisms of prism adaptation (PA) in patients with left-brain damage is on record, and about putative differences of PA, and the aftereffects (AEs), between patients with left and right brain damage. In the present study, PA and the AEs were assessed in 30 brain-damaged patients, 20 with right-sided lesions (10 with and 10 without USN), and 10 with left-sided lesions without USN, as well as in a control group of 24 age-matched participants. All patients underwent adaptation to lenses shifting the field of vision towards the side of the lesion, followed by two measures for detecting AEs: the proprioceptive (P) and the visuo-proprioceptive (VP) straight-ahead tasks. To investigate the temporal course of AEs in the different groups, the two measures were recorded immediately and 10 min after PA. Before PA, and at the end of the 10-min delayed evaluation, two tasks to assess USN (target cancellation and drawing) were also administered. All patients adapted to prisms. However, left-brain-damaged (LBD) patients presented with reduced AEs, as compared with right-brain-damaged (RBD) patients with USN. Moreover, while both controls and LBD patients adapting to left-shifting prisms had reduced VP AEs in the delayed condition, AEs were not different from zero (i.e., no AEs) in LBD patients. Finally, in the delayed condition USN patients showed an improvement in the drawing, but not in the cancellation, tasks. These results suggest that adaptation to leftward shifting lenses is associated with larger decay of VP AEs, and a role of the left hemisphere in maintaining these AEs after PA. These findings can be of relevance for the clinical application of this technique in neurological populations.