There is growing evidence that both the basal ganglia and the cerebellum play functional roles in emotion processing, either directly or indirectly, through their connections with cortical and subcortical structures. However, the lateralization of this complex processing in emotion recognition remains unclear. To address this issue, we investigated emotional prosody recognition in individuals with Parkinson’s disease (model of basal ganglia dysfunction) or cerebellar stroke patients, as well as in matched healthy controls ( n = 24 in each group). We analysed performances according to the lateralization of the predominant brain degeneration/lesion. Results showed that a right (basal ganglia and cerebellar) hemispheric dysfunction was likely to induce greater deficits than a left one. Moreover, deficits following left hemispheric dysfunction were only observed in cerebellar stroke patients, and these deficits resembled those observed after degeneration of the right basal ganglia. Additional analyses taking disease duration / time since stroke into consideration revealed a worsening of performances in patients with predominantly right-sided lesions over time. These results point to the differential, but complementary, involvement of the cerebellum and basal ganglia in emotional prosody decoding, with a probable hemispheric specialization according to the level of cognitive integration.