In recent years, there has been increasing evidence of cerebellar involvement in emotion processing. Difficulties in the recognition of emotion from voices (i.e., emotional prosody) have been observed following cerebellar stroke. However, the interplay between sensory and higher-order cognitive dysfunction in these deficits, as well as possible hemispheric specialization for emotional prosody processing, has yet to be elucidated. We investigated the emotional prosody recognition performances of patients with right versus left cerebellar lesions, as well as of matched controls, entering the acoustic features of the stimuli in our statistical model. We also explored the cerebellar lesion-behavior relationship, using voxel-based lesion-symptom mapping. Results revealed impairment of vocal emotion recognition in both patient subgroups, particularly for neutral or negative prosody, with a higher number of misattributions in patients with right-hemispheric stroke. Voxel-based lesion-symptom mapping showed that some emotional misattributions correlated with lesions in the right Lobules VIIb and VIII and right Crus I and II. Furthermore, a significant proportion of the variance in this misattribution was explained by acoustic features such as pitch, loudness, and spectral aspects. These results point to bilateral posterior cerebellar involvement in both the sensory and cognitive processing of emotions.