Working Memory (WM) is a cognitive system whose crucial role is to temporarily hold and manipulate information. Early studies suggest that verbal WM is typically associated with left hemisphere (LH) brain regions, while the processing of visuospatial information in WM more specifically depends on the right hemisphere (RH). However, recent evidence suggests a more complex network involving both hemispheres' prefrontal and posterior parietal cortices in these processes. Unfortunately, previous lesion studies often examined only one modality (either verbal, or visuospatial) or one hemisphere, which limits the possible conclusions regarding non-lateralized hemispheric involvement. Using connectome-based lesion-symptom mapping on a large sample of patients with left (LBD) and right (RBD) focal brain damage, we examined whether gray matter damage and white matter disconnections predict deficits of WM updating in an N-back task. Patients were examined with two WM tasks that differed regarding modality (verbal, spatial) and cognitive load (1-back, 2-back). Behavioral outcomes indicated that RBD patients showed significant deficits in WM updating, regardless of task modality or load. This observation was supported by whole-brain voxel-based analysis, revealing associations between WM deficits and gray matter clusters in the RH. Specifically, damage to the right lateral frontal cortex including the brain region homologous to Broca's area was associated with verbal WM deficits, while damage to the right inferior parietal lobe and posterior temporal cortex predicted spatial WM deficits. Additionally, white matter analyses identified severely impacted tracts in the RH, predicting deficits in both verbal and spatial WM. Our findings suggest that the mental manipulation of both verbal and visuospatial information in WM updating relies on the integrity of the RH, irrespective of the specific type of information held in mind.