Category and letter fluency tasks have been used to demonstrate psychological and neurological dissociations between semantic and phonological aspects of word retrieval. Some previous neuroimaging and lesion studies have suggested that category fluency (semantic-based word retrieval) is mediated primarily by temporal cortex, while letter fluency (letter-based word retrieval) is mediated primarily by frontal cortex. Other studies have suggested that both letter and category fluency are mediated by frontal cortex. We tested these hypotheses using voxel-based lesion symptom mapping (VLSM) in a group of 48 left-hemisphere stroke patients. VLSM maps revealed that category and letter fluency deficits correlate with lesions in temporal and frontal cortices, respectively. Other regions, including parietal cortex, were significantly implicated in both tasks. Our findings are therefore consistent with the hypothesis that temporal cortex subserves word retrieval constrained by semantics, whereas frontal regions are more critical for strategic word retrieval constrained by phonology.