Right hemisphere lesions often lead to severe disorders in spatial awareness and behavior, such as left hemispatial neglect. Neglect involves not only pathological biases in attention and exploration but also deficits in internal representations of space and spatial working memory. Here we designed a new paradigm to test whether one potential component may involve a failure to maintain an updated representation of visual locations across delays when a gaze-shift intervenes. Right hemisphere patients with varying severity of left spatial neglect had to encode a single target location and retain it across an interval of 2 or 3 sec, during which the target was transiently removed, before a subsequent probe appeared for a same/different location judgment. During the delay, gaze could have to shift to either side of the remembered location, or no gaze-shift was required. Patients showed a dramatic loss of memory for target location after shifting gaze to its right (toward their "intact" ipsilesional side), but not after leftward gaze-shifts. Such impairment arose even when the target initially appeared in the right visual field, before being updated leftward due to right gaze, and even when gaze returned to the screen center before the memory probe was presented. These findings indicate that location information may be permanently degraded when the target has to be remapped leftward in gaze-centric representations. Across patients, the location-memory deficit induced by rightward gaze-shifts correlated with left neglect severity on several clinical tests. This paradoxical memory deficit, with worse performance following gaze-shifts to the "intact" side of space, may reflect losses in gaze-centric representations of space that normally remap a remembered location dynamically relative to current gaze. Right gaze-shifts may remap remembered locations leftward, into damaged representations, whereas left gaze-shifts will require remapping rightward, into intact representations. Our findings accord with physiological data on normal remapping mechanisms in the primate brain but demonstrate for the first time their impact on perceptual spatial memory when damaged, while providing new insights into possible components that may contribute to the neglect syndrome.